SharePoint Data Governance: Achieve Consistent & Automated Security Webinar Invite

Next Friday September 30 I will be co-presenting a live webinar on SharePoint Data Governance with Microsoft and our partners Titus. I will be speaking to some real world examples of handling SharePoint security and regulatory compliance challenges using SharePoint and Titus Metadata Security.

The amount of data - and sensitive data - is growing within SharePoint environments every day, but is your organization set up to keep it all secure?

Date: September 30, 2011
Time: 12:00pm EST
Speakers:
Dr. Soheil Saadat
Principal Program Manager
Microsoft

Nick Kellett
Microsoft MVP & CTO
StoneShare

Antonio Maio
Senior Product Manager
TITUS

Register Now!

You can find out more information on our StoneShare.com website here:

http://www.stoneshare.com/news/Pages/TitusLabsWebinar.aspx

SEO Tool for SharePoint 2010 WCM Sites

There is a great tool from Mavention called Meta Fields that allows SharePoint 2010 administrators to add metadata properties to the source code of their pages. This makes it easier for a search engine to crawl and rank the content and so is helpful for Search Engine Optimization (SEO).

To use this you just have to download (for free as far as I can tell) and install Mavention Meta Fields which is a SharePoint solution. After activating the feature and adding metadata columns to your page content type, you can now edit the properties and the source code spits out metadata such as (for example) keywords, subject, DC.title and DC.keywords.

Anyone with a public website SharePoint 2010 web site should consider using this. Complete instructions are on creator Waldek Mastykarz’s blog:

http://blog.mastykarz.nl/easy-editing-meta-tags-publishing-pages-mavention-meta-fields/

Great solution Waldek!

Managed Metadata Term Set Tools

One of the great new features of SharePoint 2010 is the new Managed Metadata service, which allows you to centrally manage your metadata. You can setup hierarchical terms in a variety of languages, and delegate the administration responsibility to end users such as Information Management staff or Records Managers.

Although it is easy enough to add individual terms, there are a variety of great tools that make it easy to create and upload entire term sets (the largest we have migrated so far is Medical Subject or MeSH data with about 16,000 hierarchical medical terms).

Here are some of the tools we use at StoneShare:

Excel Template with Macro

Wictor Wilen has created an Excel Template with a Macro to make it easy to populate SharePoint 2010 Term Sets.

I have uploaded the file to the Agora Development site under Shared Documents > Development Tools: https://agora.stoneshare.com/dev/Shared%20Documents/Development%20Tools/TermStoreCreator.xltm

The instructions are located here:

http://www.wictorwilen.se/Post/Create-SharePoint-2010-Managed-Metadata-with-Excel-2010.aspx

Term Set Importer / Exporter

When uploading large term sets (such as the MeSH one we used for a client) you might get timeouts in Central Administration – this is because the out-of-the-box import tool uses a web interface). There is a great (FREE!) tool on CodePlex which uses a little desktop utility, and hence does not time out:

http://termsetimporter.codeplex.com/ 

Pre-Built Term Sets for Sale

Rather than going to the trouble of creating your own term sets, you can also purchase existing term sets from Data Facet. I haven’t yet used these so I can’t speak for them but this could be a good way to quick-start your managed metadata for a particular industry.

http://www.datafacet.com/sharepoint.aspx

One advantage of this approach (apart from the time saved recreating these) is that you will be using the identical term set as other organizations, which may help with data portability. Whether this is your business requirement or not is an open question.

I hope these tools help!

When SharePoint 2010 Met Web 2.0

One of the goals in SharePoint 2010 was to make it easier for users to update their information and pages without lots of postbacks, clicking, and delays. Accordingly, Microsoft has invested a lot in improving the web user interface.

One way they have done this is by adding the Office Ribbon concept to SharePoint. I think this has to be a first for a web application, and to be honest while I saw the value in Office 2007, I wasn’t sold on it for a web interface.

I think the major weakness of the Ribbon concept is that you can spend a fair amount of time trying to remember what command belongs to what tab. As well, it doesn’t always save clicks. More on that in a moment.

The other major investment Microsoft made is adding AJAX. This is  no-brainer and a hands-down winner for me. I’ve attached some screenshots to show how you would modify a page in the new UI.

Let’s imagine you want to modify a team site:

Step 1: You are in the Browse tab of the Ribbon (up top) – choose the Edit Tab.

To Edit, click “Edit” which is one of the buttons on the Edit tab. Then click on the area of the page you want, type some text in, and click Stop Editing. Are we saving clicks yet? :)

Well, not so far, but there weren’t any postbacks, so overall I think there’s some time saving here. An important benefit from a training perspective is the server and office products now have identical user experiences, which is a big win.

As well, there are some nice new options including an XHTML converter. And did I mention this all works flawlessly in FireFox? Web standards, hooray!

You can also insert new web parts via the Insert section of the Edit Ribbon:

Of course, the context-based Ribbon experience continues when managing lists and libraries. Here’s a screenshot of the out of the box Shared Documents library’s two important ribbons, Documents and Library:

Finally, tagging and sharing is a major concept in Web 2.0 and SharePoint 2010 addresses this by surfacing sharing activities through the Ribbon. Content can be easily tagged - Tags can be private or public and are automatically added to a suggested set so that users can share tags.

Tagging is also part of a user’s Activity Stream (not sure what the official term is). You can see on my profile that I tagged an element.

I’m not showing it here but there is also an Enterprise Metadata service that allows an organization to centrally control its taxonomy. So, now you can make peace between the “folksonomy” and “centralized taxonomy” gangs in your office!

All in all these UI improvements are icing on the SharePoint 2007 cake. I’m not sure they are enough by themselves to encourage SharePoint 2007 customers to upgrade (I think there are better reasons to upgrade), but somebody with 2003 or without SharePoint at all might now make the plunge. However, these are welcome additions to an already great product.

Although I’m not convinced the ribbon will save clicks, and will certainly take some retraining and familiarization time, it at least is consistent with the Office clients, making for tighter integration. The AJAX-style UI is a big win, and the inclusion of some interesting tagging and sharing features brings SharePoint up-to-date with the Web 2.0 world.

No More Metadata Migraines: Easily Manage Your Centralized Taxonomy

Well, I just finished my preso, which seemed to go very well (luckily!).

Basically my session covered best practices for taxonomy in SharePoint, including things you need to do (or not do) to make your lives easier when managing your portal's content.

It also covered ways you can leverage your metadata using Content Query Web Parts, property searches, and advanced searches.

Mindsharp will probably be making the presentation available on their SharePoint Best Practices website shortly. In the meantime I've uploaded the presentation in PDF format here: 

http://www.echotechnology.com/Events/Documents/No%20More%20Metadata%20Migraines%20-%20echoTechnology%20-%20SharePoint%20Best%20Practices%20Conference.pdf

As always I'd love to hear back from you about what your best practices are. My email's in the PDF, you can contact me via LinkedIn (http://www.linkedin.com/in/nicholaskellett) or you can comment on the blog.

The conference is over today, and we've really enjoyed it. We had lots of great conversations in seminar rooms, around the lunch table, and around the bar. It was neat to meet so many people who are enthusiastic about SharePoint, trying to do the right things with it, and willing to share their knowledge. I'll be blogging a bit more in the next few days, to try to transmit what I've learned.

Finally, congratulations to Mark Elgersma, Ben Curry, Bill English, and all the other Mindsharp folks who worked so hard to pull this off!

SharePoint Metadata Best Practices

I'm in the middle of writing the first of several white papers for echoTechnology. echoTechnology is keen to provide prescriptive guidance when clients are making sweeping portal changes using its echo for SharePoint product, and this is part of that ongoing effort.

The goal of this particular white paper, on best practices for managing SharePoint metadata, is to help provide a series of specific DO's and DON'T's to help SharePoint Administrators when they are deploying and managing content types and site columns in their SharePoint farms.

I've come up with the following so far:

Goals

• Support the creation of a Single Source of Truth in the portal – where content exists in only one place but can be displayed in multiple places
• Create a consistent taxonomy for the portal
• Centralize the management of this taxonomy
• Ensure this taxonomy can be applied across the portal and across all the environments
• Ensure this taxonomy can be updated as business requirements change
• Users should be able to inherit the core taxonomy and enhance with their particular needs
• Any changes make should be shared with the rest of the business, to reduce duplication

High Level SharePoint Requirements

• Identification and creation of core content types for entire portal
• Centralization of core content types in root of each site collection
• Ability to push centralized content types across the whole portal 
• Centralized content types must be updatable and any modifications can be pushed down to the subsites
• Advanced Search should allow the metadata properties to be specifically targeted
• Content Query web parts should be used to surface this content across the portal using the metadata
• Developers should be able to programmatically use the custom content types without running into problems
• Users must be educated and helped to use these content types and site columns
• Users are still able to apply their own metadata when required by inheriting and adding custom columns

The specific Best Practices in SharePoint I recommend at the moment are:

Creating and Managing Content Types

• Create the content type at the top level of a site collection. This ensures it is visible and can be used across the site collection.
• By default, your content types will be placed in the "Custom Content Types" group. To help manage multiple content types over time, we recommend that you create your own group. Name it after your organization, or give it some other meaningful name.
• Create a “root” content type for each type of Parent Content Type that you plan to use. A root content type is a custom content type that inherits directly from one of the Out-of-the-Box Content Types but that won’t be used directly in the portal. To avoid confusion, give the root content type the name of the parent content type, prefixed or suffixed by “Root”. All of your custom content types should inherit from a root content type. By creating these up front, you reserve the ability to later add one or more columns that all the children content types will automatically inherit. This powerful feature will reduce duplication and help ensure consistency across your content types.
• Create a custom group name for your content types. Name the group after your organization or give it some other meaningful name. This is a useful way to filter content types as the portal grows.
• When adding a new site column to one of your root content types, choosing to “Update list and site content types” will propagate your changes to any inheriting child content types. This is desirable behavior since any changes you make at the top level are by definition required across the business. If you do not wish to push down these changes to child content types, then you should perhaps rethink why you wish to add that column to the root level.
• Don’t create too many core content types. Users will have to pick amongst these, and the more there are the more confusion will set in. Target the 80% of common use cases and let your users create their own child content types to satisfy any additional requirements.
• Don’t include too many metadata columns in your content types. The more effort users have to expend while saving documents, the more dissatisfied they will be. Forrester Research quote: “It was so unusable. Eleven metadata fields! We just stopped using it altogether and started managing our documents on our workstations, another file share, anything to avoid having to use this system.”[1]
• Only create site columns for metadata information that is not likely to be contained in the document itself, and that you intend to leverage. If you won’t be targeting a particular piece of metadata via the Advance Search Properties or via a Content Query Web Part, then don’t force users to fill it in. Later, if you find you need to target these properties, you can add them to the parent content type and push the change amongst all the children.
• However many fields you use, make it as easy as possible for users to populate these. Set smart default choices, and make as many fields optional as you can.
• In the case of Word 2007 documents, you can consider creating a custom Document Information Panel that makes it easy to enter metadata and provides filters and lookups behind the scenes. You can also consider embedding a custom workflow in your content type. It can be set to start when creating a new item, or when the item is changed. When triggered, it can fill in more metadata behind the scenes, perhaps using lookup variables such as the user name and department.
• Use the Link to a Document content type whenever you have content stored elsewhere on the portal or in any other location accessible by a URL. This supports the goal of having a Single Source of Truth. Many organizations use this as part of a system of stub references, which link a front end SharePoint portal with their back-end Document Management systems.

Creating and Managing Site Columns

• Create the site column at the top level of a site collection. This ensures it is visible and can be used across the site collection.
• When adding Site Columns, first try to add from existing site columns. They cover a variety of needs from tracking the Author of an item to a person’s Job Title. It is worth checking whether a similar column already exists. Be aware that several especially common pieces of metadata such as Keywords and Subjects already exist and can be used.
• Description already exists but cannot be found in the list of site columns. Other reserved site columns including Title and Description are kept in a site column group named “_Hidden”. Do NOT rename out of the box Site Columns!
• When possible, favour the Choice site column type which displays a list of values as dropdown lists, radio buttons, or checkboxes. This list will reduce the chance of user error and make your metadata values more consistent.
• Don’t Repeat Yourself – leverage existing metadata such as department and project names from lookups and business data catalogue fields. This ensures when things change, your administrators do not have to modify the values in many locations (which leads to errors and inconsistencies). Because Choice columns usually provide a pre-defined list of options, they are often a good candidate for lookups and business data catalogue fields. Always check to see if the Choice values can be linked to some other centralized source, such as Business Data or list items.
• Lookups and Business Data Catalogue columns are also useful to help delegate list management responsibility. Any user with create or modify permissions on the list, or the equivalent permissions in the BDC application, will be able to adjust the values in the site column without requiring the help of a SharePoint Administrator.
• Do not create Choice values that are too specific – you risk forcing your content authors to “misfile” their content. The file folder is a classic example of a system that causes this sort of grief. If an item could conceivable belong to multiple categories, use the Checkbox Choice column type.
• Create a custom group name for your site columns. For consistency, give it the same name as with the Content Types (above). This is a useful way to filter site columns as the portal grows.
• Take care when renaming Site Columns! SharePoint will store the original name as the Internal Column Name, but display the new column name. This internal column name is also known as the SharePoint Field Name. SharePoint will automatically escape the column name when creating the Site Column. David Hunter provides more information on common escape characters on his blog http://www.davehunter.co.uk/Blog/Lists/Posts/Post.aspx?List=f0e16a1a-6fa9-4130-bcab-baeb97ccc4ff&ID=95[2]. Michael Markel identifies a further limit to SharePoint internal naming: the name cannot be more than 32 characters long[3].
• As a best practice, create a Site Column with a name that contains no special and is no longer than 32 characters. Once you’ve created it, you can rename the column’s Display Name using the special characters and make it any length you like.
• An easy way to find the Internal Column Name is to view the Site Column in the Site Columns Gallery. Its URL contains a Column parameter which is the Internal Name.
• NEVER rename the Title column, as this is a reserved name! Doing so will change the Title display name all over the portal, and attempting to change it back will result in this error: “The Column name that you entered is already in use or reserved”. If you wish to change the appearance of a Title column, you should create your own custom site column and hide the original Title column in all views.

Your Advice, Please!

Because these are best practices recommendations, I would love to have your help:

1. Am I recommending anything that you feel is wrong?
2. What am I missing?
3. How do you centralize and manage your metadata in SharePoint?

Feel free to comment, and thanks in advance!

---

[1] “How To Drive Document Management Adoption”; Kyle McNabb; Forrester, 2006-09-07; p 4.

[2] “Tip - SharePoint 2007 Escaped Column Names”, David Hunter, “Thinking out aloud - Dave Hunter's SharePoint Blog” ; 2008-04-06; http://www.davehunter.co.uk/Blog/Lists/Posts/Post.aspx?List=f0e16a1a-6fa9-4130-bcab-baeb97ccc4ff&ID=95

[3] "Chopped Column Names in Sharepoint”; Michael Markel; Blog; 2007-09-20; http://www.michaelmarkel.com/2007/09/chopped-column-names-in-sharepoint.html

Eating My Own Dog Food: Photo Metadata Advice

David Marsh had a great comment on my previous blog entry. I'm reposting it in its entirety because I think it is a great tip for anyone trying to manage their photos on their file system.

I did some considerable research to find the best approach for tagging Photos and videos. One concern I had was tagging photos and storing those tags in a propriety format that I would not be able to see my tags on a photo in the future and secondly ensuring that whatever program I used was using current industry standards for tagging. So with that in mind I found out that the XMP format from Adobe is becoming the defacto standard for storing metadata about photos and Microsoft have also adopted this format. The problem with most programs out there is they maintain keywords and tags in a separate database to the actual files. (Picasa does this). That is bad because if I copy all my photos to another computer or have them backed up somewhere else, without the database I have no tags. The XMP format embeds the keywords and tags in the files themselves so they are fully portable and the keywords are never lost and are program independent. So I looked at some of the Adobe photo cataloguing applications but finally chose Microsoft Expression Media 2 because of its great support for tagging photos and then embedding them in the files and also the simple extensible scripting mechanism they have for allowing you to rename and tag your photos in bulk based on the date the photo was taken, or the file name or any other piece of metadata you can think off. Very powerful, and I know my photos are storing all my keywords and tags in XMP format within the photos themselves and it is a standard that will guarantee I can read the tags on computers in 10 years or so. I think the best option at the moment.

Here are some more insights into the topic:

Jon Udell at Strategies for Internet Citizens http://blog.jonudell.net/2007/02/14/truth-files-microformats-and-xmp/

and Geoff Coupe's Blog:

http://gcoupe.spaces.live.com/blog/cns!6AA39937A982345B!4417.entry

The general consensus is that keeping the metadata with the file is the preferred approach, which makes perfect sense and is in keeping with the rest of my SharePoint-based Information Management plan.

David recommends Microsoft Expression Media 2. It also has an archive feature - the ability to backup to any mounted drive. This would work well with my JungleDisk mounted drive.

The software is $200 but I'm committed to getting organized so I'm willing to pay (a fair bit) for that if it saves me time down the road. More information on Microsoft Expression Media 2 is on the official website here:

http://www.microsoft.com/expression/features/Default.aspx?key=media

There is a trial version available here: http://www.microsoft.com/downloads/details.aspx?FamilyId=CD359E7D-FD27-4901-BAFF-6D564CFBD700&displaylang=en

For some reason it doesn't mention how long the trial lasts for. I'll download it today and see.

Has anyone else used Expression Media yet, and if so what do you think of it? If not, what other software would you use to do the job?

Eating My Own Dog Food: File Cleanup

I'm in the process of cleaning up my files, as I mentioned last week. It's going a little better than I expected.

To filter duplicate files I'm using a free tool, Easy Duplicate Finder which is available at http://www.easyduplicatefinder.com/download.html.

It does a good job of matching possible duplicates using file size, file type, and file name checking. It doesn't apply any logic to removing them, which is a good thing - I'm forced to check each possible dupe as I go and delete (or not!).

The tricky bit is figuring out which folder structure I should be using in the Hub folder...going forward I will be hosting most of this information in the cloud so I can apply labels and metadata at that point, and surface these files in all sorts of funky ways, but right now I still need a place for my files to live.

I'm committed to the whole Single Source of Truth practice and so I'm forcing myself to use Windows Desktop Search, Vista folder bookmarks, and a central desktop folder root.

It's been a real question of discipline - I often find myself reverting to bad practices and trying to copy the same file in multiple places if I can't quite decide where it should go. Over the last couple of days I've started getting a little better at placing any particular file in a single place and adding shortcuts and bookmarks to it.

Cleaning Up Music Files

• I did a desktop search for *.mp in the root of the old c:\music folder
• I copied all the files which were the results of the search into a new, single folder called "cleanup"
• Then I did a duplicate file finder scan on that folder so it could clean it all up

 

• I deleted the duplicate files...and went from about 3,920 files and 12 gigs in the music folder to about 2,152 files and 8.9 gigs of music.

Then I foolishly decided to let iTunes manage the folder.

• I set the iTunes folder to be hub\music folder and made sure Create file with trackname was disabled as this was causing duplicate files all the time
• I added the cleanup folder to iTunes
• I allowed iTunes to consolidate the library which organized the files which renamed them and placed in the hub folder
• Now I had a lot of duplicate tracks, so I followed this advice to remove those:
• http://www.tech-recipes.com/rx/1579/how_to_remove_duplicate_files_from_your_itunes_media_library_

For some reason no matter what I do iTunes duplicates the files 2 to 4 times. As near as I can tell, it is doing this because it thinks the song could belong to multiple albums for that artist....Whatever the case, I am left with iTunes havoc and I am still puzzling out how to remove duplication in this folder without deleting original songs.

Thanks iTunes! Maybe it's not worth the bother.

Cleaning Up Photos and Videos

• I did a duplicate file clean on the photos and videos folder
• This is somewhat harder than managing music - as most cameras will tag a file with the same broad tag and then you copy 100 pictures labeled [Tag] 001, [Tag] 002, etc...
• It required lots of manual passes, and I made sure not to delete things I wasn't sure about
• I did some targeted searches using Windows Desktop search tool - looking for keywords and trying to merge things into broader folders
• In the end, I've gone from 2100 files (including 1102 duplicates) and 36 gigs of content to 998 files and 28.7 gigs of space.

Metadata is much more important for photos than for music files, because I can use them in many more ways (image editing, personal scrapbooks, social networking sites, websites).

Because an image can be used in many different places, it isn't easy to choose a single folder for a photo to live in. I've started experimenting with bulk insert into Picassa or similar so I can tag 'em. Also pondering adding del.icio.us to the Information Management plan since I am in the whole Cloud frame of mind.

How do you manage photos on your desktop, especially with metadata? Are there photo management tools you couldn't live without?

Data In, Data Out

An important emerging web development initiative concerns "data portability", or how openly and transparently a website or application consumes and provides data. In a world of mashups, web services, and walled gardens like Facebook or MySpace, it is becoming an increasing concern for a variety of reasons.

To begin with, users want to know that the information they enter into a website is theirs to manage, share, and remove over time.  This matters for convenience (so they don't have to reenter it all the time) but also from a privacy perspective.

A famous recent example saw Robert Scoble temporarily banned from Facebook for breaking its terms of use by running a script to access his social graph (contacts). This focused a great deal of attention and debate on who "owns" this sort of information - the person entering the data, the website that hosts it, or the individuals whose information is being stored.

Developers also care about this as we are all tired of reinventing the  wheel every time we want to share data with another application. A major consequence of the recent wave of web development trends is the increasing importance of application integration and data integration standards. These days, "No Web App Is An Island".

To help address these needs, a community is coalescing under the banner of the DataPortability Group. Their website is located at http://www.dataportability.org/ and discusses the issues in detail. They define portability as

both physically moving data or simply porting the context in which the data is used

Their effort involves identifying and evangelizing existing data portability standards, rather than creating new ones. They also hope to encourage a trust framework that will benefit vendors and consumers.

The rapid adoption of this initiative shows how quickly things move in the IT world, if proof were needed. The project was first founded on October 11, 2007, and is already gaining a great deal of steam. The website has an interesting timeline showing how quickly support is building: http://groups.google.com/group/dataportability-public/web/buzz . 

Some recent events of note included Google, Plaxo, LinkedIn, Twitter, Flickr, Six Apart and Facebook joining the workgroup; Google, Yahoo, IBM, Microsoft, and VeriSign joining the OpenID Foundation board; and MySpace launching its Open Developer Platform.

Obviously most of the activity is driven by the Social websites as they have the most to win / lose. Issues of personal privacy and trust are crucial to their continued popularity (the Facebook Beacon PR disaster is a prime example of this). 

Nonetheless, I imagine within a few years every major Software vendor will have the DataPortability-compliant tickbox in their sales material, or their shareholders will demand to know why not. Case in point: Blogger, owned by Google, now supports OpenIDs on its blogs (such as this one), and Yahoo users can use their Yahoo! accounts as OpenIDs.

Interestingly enough, some of the standards the DataPortability group advocates are the ones that might lead to the long-envisioned semantic web - namely microformats, which can potentially add machine-readable "context" to data.

It's an interesting space to explore, and I plan to blog about it as I learn more.

SharePoint Migration With MetaLogix

I recently had the chance to evaluate a number of SharePoint content and site migration tools. Past migrations I've done have been performed using the out-of-the-box content migration using the Pre-Upgrade scan, but I have an upcoming requirement to leave the existing SharePoint Portal Server 2003 site and content in place and migrate it slowly to the new MOSS 2007 server over time.

This required an evaluation of third-party site migration tools. One of the ones I looked at was the MetaLogix SharePoint Site Migration Manager. The evaluation license allows you to migrate 50% of the content for a few days, which is enough to run the software through its paces.

The Migration tool installs on a client desktop and this GUI handles 95% of the migration tasks. However if you want the migration process to do things like map the user accounts you'll also need to install a special service on the target server.

I liked MetaLogix Migration Manager's intuitive interface. After the installation it was easy for me to add a reference to both the test SPS 2003 portal and the test MOSS 2007 portal. This is done by typing in the URL and either using the default portal access account or manually specifying it.

The interface has three major views:

1. The default "Explorer" view, which is a complete site hierarchy view of the sites you have added;
2. "Browser" view, which renders whatever site node you have selected in the explorer view. It's a great way to quickly view the content of a site or list; and
3. "Item" view, which shows you the content of lists and libraries.

Migrating content or sites between portals was as easy as right-clicking the source node, clicking "Copy", and right-clicking on the target site and clicking "Paste". A dialog box shows the progress and the result appears in the Log window at the bottom of the screen, which you can always refer to. There is verbose logging in a text file.

Other nice features:

• Batch mode, so that all actions can be scripted to run at scheduled times. The batch file is in XML format so it can be programmed and is human-readable.
• Delta migration: You can choose to only migrate the changesets instead of all the content each time.
• List copy: You can choose to append and concatenate multiple libraries or lists together which helps merge them in the new portal.
• You can migrate versioning and security permissions
• Replace Within Field: This is a quick string search that allows you to replace text in content you've migrated, such as URLs. In the future I'm told this will be an automatic step in the migration.
• Mapping to site templates. You can explicitly map SharePoint 2003 site templates to the new MOSS templates (it will attempt to map them automatically by default). I noticed the mapping options seemed to include the Fantastic 40 site templates which was a nice touch.

The only issue I found with the product occurred when I tried to migrate the content on an SPS 2003 Area over to MOSS 2007. It brought all the content, web parts, lists, and libraries over with the exception of the Area Detail and Browse Area By web parts. However as those two web parts relate to the old SharePoint 2003 Area concept which no longer applies, I'm not too fussed. The MetaLogix staff were keen to help me investigate this, if I had been willing.

During the course of the trial I spoke to Julien Sellgren and Rasool Rayani at MetaLogix. They were helpful and responsive to my questions, giving me a walkthrough via a webcast and extending the evaluation period while I was testing area migration. Rasool also explained to me that they are releasing feature packs and updates on a 6-week basis so the product is evolving quickly.

You can learn more about MetaLogix and download the evaluation version at their website.

Yesterday evening I saw a live demo of echoTechnology's echo for SharePoint 2007 product, which is a tool for handling SharePoint migration and management. The presentation was
given by CEO Garry Smith, Technical Director Sergio Otoya, and Technical Evangelist Stephen Cummins. The company is positioning echo as the tool that "allows administrators to
effectively manage and migrate onto the SharePoint 2007 platform". The demo was an hour long and was really impressive. Following that I spoke with Stephen to get a little more
information about how they feel the product can be used. I'll be publishing that within a few days.

Last night's demo gave a clear indication of echo's design philosophy, which is to make replication and change management as easy and granular as possible. This is important
because it's actually really hard to roll changes across different SharePoint versions, sites, and environments. I've mentioned before on this blog how complicated I find it to
properly manage the customizations that I make. Echo aims to solve that.

The two core concepts or "streams" that echo specializes in handling are migration and management. Migration might include migrating content from SharePoint 2003, Domino,
Exchange, or file shares. Management includes managing workflows, features, web parts, content types or site hierarchies. By creating discrete "tasks" for each of these, echo
gives you very granular control, allowing you to do one or more tasks handling simple or complicated scenarios.

The first part of the demo was a migration from SharePoint 2003 to SharePoint 2007. Out of the box SharePoint provides options such as in-place, gradual, and content migration.
The steps required to do these, and the pros and cons of each, are well documented elsewhere on the net so I won't go in to them here.

The way echo does it is to use a fresh install of SharePoint 2007 as a clean environment in which to push the 2003 migration. You first create a template site in MOSS 2007 (they
called it the "Blueprint site") which has all the webparts, lists, and views you would like to migrate your 2003 stuff to. Next, you select the 2003 sites you would like to
migrate. The lists of migration candidate sites get added to what looks like an Excel plugin - this allows you to save the list to Excel and give it to your business users so
they can make any changes they want pre-migration. This ensures that someone can clean up metadata, illegal characters, or make any other changes before you do the site
migration. You can run the job right away, or schedule it to run automatically at a later time. You can also save it to use for other batches. Once the batch ran, we saw how the
sites from 2003 were all copied over to the 2007 portal. A log gives complete details on each step of the migration.

After the run, the web parts and lists were migrated, but not any of their content. This led to the second demo, which was about content migration or "content loading" as it is
called. Having migrated the 2003 sites over, we were shown how the Content Loader Task allowed an administrator to map existing 2003 content, including its related metadata,
over to content types created on the 2007 portal. The mechanism to do this is very granular, so content settings, fields, views, permissions, version history and the like can
all be migrated across in whatever way you want.

There is even a choice to truncate the version histories, so that if you wanted you could migrate only the last couple of versions of content to the new 2007 portal. Once again
all of this can be saved to an Excel "control file", which allows customers to modify the migration settings if they want. This allows a deep level of control over the process,
and the Excel format is very user-friendly. In fact if I understood correctly formulas could even be run over the control files to do quick formatting.

The echo team say that one of the primary goals of the Content Loader Task is to help manage content types and support a central taxonomy. They feel that this is a key
requirement to really gain value from a MOSS portal with targeted searches. They have seen that many organizations don't bother doing this in a systematic way due to the adhoc
and unstructured way SharePoint 2003 allowed metadata columns to be added. Since an out-of-the-box migration will just try to copy existing metadata columns across without any
adjustments, and since the administrators doing the migration don't have any knowledge of what metadata should be placed on the content, the tendency is just to port all of
it over to the 2007 portal without applying any lessons learned or disciplined taxonomy. By using the Excel control file, the burden of applying appropriate metadata is removed
from the shoulders of the administrators handling the migration, and placed in the hands of content authors where it belongs. The Excel format seems like a logical way and
friendly to get content authors to plan and manage their metadata before a migration.

Having demonstrated the ease with which metadata can be changed in the Excel control file, it was reloaded into the echo interface, the batch was run, and all the content
migrated over to the 2007 sites and mapped to Content Types as planned.

The next demonstration was a quick example of loading content from file shares - in this case images kept in a series of folders. What was neat here was that there were four
folders with only two duplicated images in each - basically an example of versioning as it is usually done on a file system. The content loader provided a mechanism to
"collapse" these files together so that upon import, there were only two images but they each had the complete version history of their four versions taken from the folders.

At this point we were shown the concept of batching, which facilitates scheduling and change management for SharePoint. Any number of tasks could be added and configured,
providing complete flexibility for whatever scenario was required. The batches could be saved for future use, and scheduled to run at a particular time (such as after business hours). Batching is also an effective mechanism for migrating changes between environments.

Things I found noteworthy:

• The Excel plugin approach is a very good idea. Garry suggested this was driven directly from customer feedback.
• The granular level of tasks is useful because no-one is second-guessing how you want to do a migration or management - you are free to chain the tasks together in whichever
  order or process you like.
• The ability to migrate workflows is pretty key as one of the drawbacks of using SharePoint Designer is that it links a workflow to a particular list - and echo would remove that
  problem.

I'm sure that some organizations will value this tool for the help it can give migrating from SharePoint 2003 to 2007, especially with more complicated sites. It may help such
organizations "jump the gap" if they are hesitating due to the complexity of managing the transition. I haven't had to do many upgrades and those have been straightforward
"content migrations" so this is less important to me. Personally I'm most interested in the potential it has for migrating SharePoint customizations from development to testing
to production. This is a requirement in any organization that supports SharePoint and very hard to address without such a tool, and it's an ongoing need.

Echo for SharePoint 2007 is scheduled for release at the end of July. If you're interested you can find out more information and download a trial at echoTechnology's website
(http://www.echotechnology.com/).

This very helpful tip comes by way of Mick Badran at Breeze Training (http://www.breezetraining.com.au/site/), who incidentally puts on a first-class MOSS 2007 Boot Camp. One great thing about SharePoint is that it can help to free people from the whole file structure "folder paradigm". Instead of trying to remember which folder something is in, people should be able to upload documents, apply reasonable metadata, and find them again using searches and lookups. That's why I often try to disable Folder creation on document libraries in the portal.

Back in the real world, keeping folders is sometimes a necessary evil. Here is an approach that at leasts adds some Information Management best practices to the process.

If you want to apply metadata to folders in SharePoint 2007 document libraries, you can create a custom content type with custom columns on it. Simply create a new custom content type at the root of your site collection, and select "Folder Content Types" from the "Select parent content type from" dropdown box, and "Folder" from the "Parent Content Type" filtered list.


Then, add your custom columns to this content type. These will be the metadata fields that appear when a user creates a new metadata folder.

Go back to the Library settings, and select "Advanced Settings". Make sure you enable the "Manage content types" option. Also remove the ability to "Allow users to create new folders". This prevents them from adding generic folders and forces them to select your metadata-driven ones.

Now on the New dropdown on the library, users can select your "Metadata Folder". Doing so will prompt them to enter the metadata you have applied to it. Folder information will be displayed in the library views, although you may have to edit the current view and select the new columns for display. Of course results will also appear in the Search Results, once a crawl has been made.