Poll: Open Source .NET Web Frameworks

Next Stop, SharePoint Best Practices Conference!

Turns out I will be attending the SharePoint Best Practices Conference in Washington in a couple of weeks. echoTechnology is one of the Gold Sponsors, so I will be helping to man the booth.

Better yet, I get to do a 1-hour demo on how echo helps manage SharePoint using best practices. Details are still being worked out, but I'm hoping to demonstrate the Centralized Metadata best practices I've been working on.

So if you're at the conference and can spare an hour, please drop by to my presentation - Wednesday at  2:20 to 3:30. Even if you can't make it - you can stop by the booth any time and say hi!

Hope to see you there,

Nick

SharePoint World Record Dash!

In honour of the Beijing Olympic Games I thought I would create a new event, the SharePoint World Record Dash. The goal is to update a blank Team Site to include a whole new look, layout, and content, in the shortest possible time.

I'm using a preset echo for SharePoint batch to run the series of changes on one site, and Adobe Captivate to film the results - I awarded myself the broadcast rights :)

The changes are:

1. Apply a new theme for the site (Obsidian)
2. Remove the existing WSS Image Web Part
3. Add an RSS Feed Viewer web part pointing to the BBC Sports news
4. Add a custom link to the BBC website on the Links list
5. Add a Content Editor web part with Wikipedia information

echo will use the SharePoint object model to make these changes, just as if I'd done them all by hand via the browser.

In this video I'm only updating one site but in my weekly webinars I show how easy it is to make identical changes in development, testing, production environments, on local or remote farms, and to multiple site collections and subsites.

Without further ado, here's the SharePoint World Record Dash video.

If you want to see the batches in more detail, feel free to sign up for one of our weekly webinars, every Thursday at 1 pm PST. You can sign up at http://www.echotechnology.com/registration/Events.aspx?PID=8&EVTID=1

Hope you like this!

SharePoint Metadata Best Practices

I'm in the middle of writing the first of several white papers for echoTechnology. echoTechnology is keen to provide prescriptive guidance when clients are making sweeping portal changes using its echo for SharePoint product, and this is part of that ongoing effort.

The goal of this particular white paper, on best practices for managing SharePoint metadata, is to help provide a series of specific DO's and DON'T's to help SharePoint Administrators when they are deploying and managing content types and site columns in their SharePoint farms.

I've come up with the following so far:

Goals

• Support the creation of a Single Source of Truth in the portal – where content exists in only one place but can be displayed in multiple places
• Create a consistent taxonomy for the portal
• Centralize the management of this taxonomy
• Ensure this taxonomy can be applied across the portal and across all the environments
• Ensure this taxonomy can be updated as business requirements change
• Users should be able to inherit the core taxonomy and enhance with their particular needs
• Any changes make should be shared with the rest of the business, to reduce duplication

High Level SharePoint Requirements

• Identification and creation of core content types for entire portal
• Centralization of core content types in root of each site collection
• Ability to push centralized content types across the whole portal 
• Centralized content types must be updatable and any modifications can be pushed down to the subsites
• Advanced Search should allow the metadata properties to be specifically targeted
• Content Query web parts should be used to surface this content across the portal using the metadata
• Developers should be able to programmatically use the custom content types without running into problems
• Users must be educated and helped to use these content types and site columns
• Users are still able to apply their own metadata when required by inheriting and adding custom columns

The specific Best Practices in SharePoint I recommend at the moment are:

Creating and Managing Content Types

• Create the content type at the top level of a site collection. This ensures it is visible and can be used across the site collection.
• By default, your content types will be placed in the "Custom Content Types" group. To help manage multiple content types over time, we recommend that you create your own group. Name it after your organization, or give it some other meaningful name.
• Create a “root” content type for each type of Parent Content Type that you plan to use. A root content type is a custom content type that inherits directly from one of the Out-of-the-Box Content Types but that won’t be used directly in the portal. To avoid confusion, give the root content type the name of the parent content type, prefixed or suffixed by “Root”. All of your custom content types should inherit from a root content type. By creating these up front, you reserve the ability to later add one or more columns that all the children content types will automatically inherit. This powerful feature will reduce duplication and help ensure consistency across your content types.
• Create a custom group name for your content types. Name the group after your organization or give it some other meaningful name. This is a useful way to filter content types as the portal grows.
• When adding a new site column to one of your root content types, choosing to “Update list and site content types” will propagate your changes to any inheriting child content types. This is desirable behavior since any changes you make at the top level are by definition required across the business. If you do not wish to push down these changes to child content types, then you should perhaps rethink why you wish to add that column to the root level.
• Don’t create too many core content types. Users will have to pick amongst these, and the more there are the more confusion will set in. Target the 80% of common use cases and let your users create their own child content types to satisfy any additional requirements.
• Don’t include too many metadata columns in your content types. The more effort users have to expend while saving documents, the more dissatisfied they will be. Forrester Research quote: “It was so unusable. Eleven metadata fields! We just stopped using it altogether and started managing our documents on our workstations, another file share, anything to avoid having to use this system.”[1]
• Only create site columns for metadata information that is not likely to be contained in the document itself, and that you intend to leverage. If you won’t be targeting a particular piece of metadata via the Advance Search Properties or via a Content Query Web Part, then don’t force users to fill it in. Later, if you find you need to target these properties, you can add them to the parent content type and push the change amongst all the children.
• However many fields you use, make it as easy as possible for users to populate these. Set smart default choices, and make as many fields optional as you can.
• In the case of Word 2007 documents, you can consider creating a custom Document Information Panel that makes it easy to enter metadata and provides filters and lookups behind the scenes. You can also consider embedding a custom workflow in your content type. It can be set to start when creating a new item, or when the item is changed. When triggered, it can fill in more metadata behind the scenes, perhaps using lookup variables such as the user name and department.
• Use the Link to a Document content type whenever you have content stored elsewhere on the portal or in any other location accessible by a URL. This supports the goal of having a Single Source of Truth. Many organizations use this as part of a system of stub references, which link a front end SharePoint portal with their back-end Document Management systems.

Creating and Managing Site Columns

• Create the site column at the top level of a site collection. This ensures it is visible and can be used across the site collection.
• When adding Site Columns, first try to add from existing site columns. They cover a variety of needs from tracking the Author of an item to a person’s Job Title. It is worth checking whether a similar column already exists. Be aware that several especially common pieces of metadata such as Keywords and Subjects already exist and can be used.
• Description already exists but cannot be found in the list of site columns. Other reserved site columns including Title and Description are kept in a site column group named “_Hidden”. Do NOT rename out of the box Site Columns!
• When possible, favour the Choice site column type which displays a list of values as dropdown lists, radio buttons, or checkboxes. This list will reduce the chance of user error and make your metadata values more consistent.
• Don’t Repeat Yourself – leverage existing metadata such as department and project names from lookups and business data catalogue fields. This ensures when things change, your administrators do not have to modify the values in many locations (which leads to errors and inconsistencies). Because Choice columns usually provide a pre-defined list of options, they are often a good candidate for lookups and business data catalogue fields. Always check to see if the Choice values can be linked to some other centralized source, such as Business Data or list items.
• Lookups and Business Data Catalogue columns are also useful to help delegate list management responsibility. Any user with create or modify permissions on the list, or the equivalent permissions in the BDC application, will be able to adjust the values in the site column without requiring the help of a SharePoint Administrator.
• Do not create Choice values that are too specific – you risk forcing your content authors to “misfile” their content. The file folder is a classic example of a system that causes this sort of grief. If an item could conceivable belong to multiple categories, use the Checkbox Choice column type.
• Create a custom group name for your site columns. For consistency, give it the same name as with the Content Types (above). This is a useful way to filter site columns as the portal grows.
• Take care when renaming Site Columns! SharePoint will store the original name as the Internal Column Name, but display the new column name. This internal column name is also known as the SharePoint Field Name. SharePoint will automatically escape the column name when creating the Site Column. David Hunter provides more information on common escape characters on his blog http://www.davehunter.co.uk/Blog/Lists/Posts/Post.aspx?List=f0e16a1a-6fa9-4130-bcab-baeb97ccc4ff&ID=95[2]. Michael Markel identifies a further limit to SharePoint internal naming: the name cannot be more than 32 characters long[3].
• As a best practice, create a Site Column with a name that contains no special and is no longer than 32 characters. Once you’ve created it, you can rename the column’s Display Name using the special characters and make it any length you like.
• An easy way to find the Internal Column Name is to view the Site Column in the Site Columns Gallery. Its URL contains a Column parameter which is the Internal Name.
• NEVER rename the Title column, as this is a reserved name! Doing so will change the Title display name all over the portal, and attempting to change it back will result in this error: “The Column name that you entered is already in use or reserved”. If you wish to change the appearance of a Title column, you should create your own custom site column and hide the original Title column in all views.

Your Advice, Please!

Because these are best practices recommendations, I would love to have your help:

1. Am I recommending anything that you feel is wrong?
2. What am I missing?
3. How do you centralize and manage your metadata in SharePoint?

Feel free to comment, and thanks in advance!

---

[1] “How To Drive Document Management Adoption”; Kyle McNabb; Forrester, 2006-09-07; p 4.

[2] “Tip - SharePoint 2007 Escaped Column Names”, David Hunter, “Thinking out aloud - Dave Hunter's SharePoint Blog” ; 2008-04-06; http://www.davehunter.co.uk/Blog/Lists/Posts/Post.aspx?List=f0e16a1a-6fa9-4130-bcab-baeb97ccc4ff&ID=95

[3] "Chopped Column Names in Sharepoint”; Michael Markel; Blog; 2007-09-20; http://www.michaelmarkel.com/2007/09/chopped-column-names-in-sharepoint.html

SharePoint Best Practices Resources

The SharePoint technology stack covers such deep waters that "best practices" are a constant buzzword these days. After the release of SharePoint 2007 everybody was scrambling to learn the platform, but now that it is better understood, people are ready to figure out not just HOW to do something in SharePoint, but increasingly WHAT and WHY.

Given this, I thought I would mention a few new best practices resources.

To begin with, Mindsharp's doing a lot of work in this area.

SharePoint Best Practices Conference

There's a SharePoint Best Practices conference in Washington this September, sponsored by Microsoft Press and Mindsharp. It runs from Monday September 15 to Wednesday September 17, and includes breakout and cabana sessions.

Although the sessions are not nailed down yet, some of them include things like:

• Taxonomy and Governance
• Best Practices for Designing, Implementing, and Supporting Enterprise Workflow
• Defining Business Requirements
• Intranet, Extranets, Internet Scenarios

They'll be led by SharePoint MVPs and authors and should provide some real "lessons from the field" for SharePoint deployments.

Cost to attend is $995 USD. More information can be found on their website at http://www.sharepointbestpractices.com/

SharePoint Best Practices Book

Related to this is another Mindsharp effort - the Microsoft Office SharePoint Server 2007 Best Practices book. From the blurb on the back:

Achieve your IT objectives with proven, best-practice guidance on using SharePoint solutions.

Get field-tested best practices and proven techniques for designing, deploying, operating, and optimizing Microsoft Office SharePoint Server 2007 and Windows SharePoint Services 3.0. [...] this guide is written by leading SharePoint MVPs and Microsoft SharePoint team members who’ve worked extensively with real-world deployments and customers. You’ll find out how to deploy the software, design your environment, manage content, analyze and view data, perform disaster recovery, monitor performance, and more. You'll learn how to create SharePoint sites that help your organization collaborate, take advantage of business insights, and improve productivity—with practical insights from the experts.

Bill English and Ben Curry wrote the bulk of the book, but as Ben points out they had

tons of help from rock stars like Kathy Hughes (MVP), Paul Schaeflein, Daniel Webster (SharePoint Guru), Mark Ferraz (IA and Security), Paul Stork, Jim Curry, and Mark Schneider (PMP).

When I left Dimension Data to move back to Canada, my team was kind enough to chip in and get me an Amazon gift certificate - and I've already used it to get this book. I should have it next week (I'm away from my house currently) and look forward to reading it and learning a lot in the process.

International SharePoint Professionals Association

There is a brand new organization called the International SharePoint Professionals Association that aims to be the umbrella organization for all SharePoint workers.

The International SharePoint Professionals Association, also known as ‘ISPA’, is an independent, not-for-profit, community-driven organization dedicated to support SharePoint professionals and groups all around the world. The primary mission of ISPA is to promote the global adoption of SharePoint Technologies by providing support and guidance to the SharePoint community as a whole – by establishing connections between SharePoint professionals, groups, resources, education and information. ISPA is led and supported by volunteers across the world, and will focus on bringing the entire SharePoint community closer together.

Although it isn't Best Practices per se, I'm mentioning this here because the organization aims to provide "support and guidance" for the SharePoint community. If it does this well, I'm tipping this could become a powerful incubator for creating and spreading best practice knowledge.

Bob Fox has the announcement on his blog. He's also got a list of regional evangelists on that post.

Follow up to "No authority could be contacted for authentication"

Anthony Moore, an IT Manager at LivePoint, provided me with some more understanding of the root causes of yesterday's demo machine errors.

Each machine in a domain has a password, (exactly like your username / password) and the passwords are reset periodically as well automatically though Active Directory...

If you restore an image of a machine to a prior state, the password in active directory becomes out of sync with [what] is stored in the machine, and the...issue can occur.

I imagine this isn't news to any Active Directory gurus out there but it clears things up for me. Going forward I will be including the AD Domain Controller in the server snapshots and rolling it back alongside the SharePoint virtual machines.

Hope this helps someone else. Thanks for the insight, Anthony!

"No authority could be contacted for authentication."

I have been giving weekly webinars using a demo environment that mimics 3 SharePoint farms (2 x MOSS 2007, 1 x SPS 2003). While prepping for tomorrow's demo I ran into some very strange SharePoint behaviour.

For no apparent reason, the default site collections no longer worked - they failed to load and couldn't see each other via a web browser. However, on both 2007 farms, I could still see and use the remote internet site collections just fine (using anonymous authentication on port 8000).

In the Event Viewer, various errors started complaining about DCOM saying things like

Reason: The trust relationship between this workstation and the primary domain failed. (Exception from HRESULT: 0x800706FD)

And when I viewed the other sites in the browser, I would get "Service Unavailable" errors and, eventually:

No authority could be contacted for authentication.

These errors, plus the fact that I could still use anonymous authentication, obviously pointed to something beyond SharePoint being the culprit. When these kind of authentication errors abound you don't have to look far beyond Active Directory to find the guilty puppy.

I eventually ran across this very helpful post by Noman Juzar Lakdawala on the ASP.NET forums:  http://forums.asp.net/t/986545.aspx

I followed his instructions and removed my servers one at a time from the domain, deleted the entry in the Domain Controller, and then readded them. Everything works now!

Not sure why AD got corrupted in the first place, so I will continue testing, but so far so good.  Thanks Noman!

Eating My Own Dog Food: Photo Metadata Advice

David Marsh had a great comment on my previous blog entry. I'm reposting it in its entirety because I think it is a great tip for anyone trying to manage their photos on their file system.

I did some considerable research to find the best approach for tagging Photos and videos. One concern I had was tagging photos and storing those tags in a propriety format that I would not be able to see my tags on a photo in the future and secondly ensuring that whatever program I used was using current industry standards for tagging. So with that in mind I found out that the XMP format from Adobe is becoming the defacto standard for storing metadata about photos and Microsoft have also adopted this format. The problem with most programs out there is they maintain keywords and tags in a separate database to the actual files. (Picasa does this). That is bad because if I copy all my photos to another computer or have them backed up somewhere else, without the database I have no tags. The XMP format embeds the keywords and tags in the files themselves so they are fully portable and the keywords are never lost and are program independent. So I looked at some of the Adobe photo cataloguing applications but finally chose Microsoft Expression Media 2 because of its great support for tagging photos and then embedding them in the files and also the simple extensible scripting mechanism they have for allowing you to rename and tag your photos in bulk based on the date the photo was taken, or the file name or any other piece of metadata you can think off. Very powerful, and I know my photos are storing all my keywords and tags in XMP format within the photos themselves and it is a standard that will guarantee I can read the tags on computers in 10 years or so. I think the best option at the moment.

Here are some more insights into the topic:

Jon Udell at Strategies for Internet Citizens http://blog.jonudell.net/2007/02/14/truth-files-microformats-and-xmp/

and Geoff Coupe's Blog:

http://gcoupe.spaces.live.com/blog/cns!6AA39937A982345B!4417.entry

The general consensus is that keeping the metadata with the file is the preferred approach, which makes perfect sense and is in keeping with the rest of my SharePoint-based Information Management plan.

David recommends Microsoft Expression Media 2. It also has an archive feature - the ability to backup to any mounted drive. This would work well with my JungleDisk mounted drive.

The software is $200 but I'm committed to getting organized so I'm willing to pay (a fair bit) for that if it saves me time down the road. More information on Microsoft Expression Media 2 is on the official website here:

http://www.microsoft.com/expression/features/Default.aspx?key=media

There is a trial version available here: http://www.microsoft.com/downloads/details.aspx?FamilyId=CD359E7D-FD27-4901-BAFF-6D564CFBD700&displaylang=en

For some reason it doesn't mention how long the trial lasts for. I'll download it today and see.

Has anyone else used Expression Media yet, and if so what do you think of it? If not, what other software would you use to do the job?

Eating My Own Dog Food: File Cleanup

I'm in the process of cleaning up my files, as I mentioned last week. It's going a little better than I expected.

To filter duplicate files I'm using a free tool, Easy Duplicate Finder which is available at http://www.easyduplicatefinder.com/download.html.

It does a good job of matching possible duplicates using file size, file type, and file name checking. It doesn't apply any logic to removing them, which is a good thing - I'm forced to check each possible dupe as I go and delete (or not!).

The tricky bit is figuring out which folder structure I should be using in the Hub folder...going forward I will be hosting most of this information in the cloud so I can apply labels and metadata at that point, and surface these files in all sorts of funky ways, but right now I still need a place for my files to live.

I'm committed to the whole Single Source of Truth practice and so I'm forcing myself to use Windows Desktop Search, Vista folder bookmarks, and a central desktop folder root.

It's been a real question of discipline - I often find myself reverting to bad practices and trying to copy the same file in multiple places if I can't quite decide where it should go. Over the last couple of days I've started getting a little better at placing any particular file in a single place and adding shortcuts and bookmarks to it.

Cleaning Up Music Files

• I did a desktop search for *.mp in the root of the old c:\music folder
• I copied all the files which were the results of the search into a new, single folder called "cleanup"
• Then I did a duplicate file finder scan on that folder so it could clean it all up

 

• I deleted the duplicate files...and went from about 3,920 files and 12 gigs in the music folder to about 2,152 files and 8.9 gigs of music.

Then I foolishly decided to let iTunes manage the folder.

• I set the iTunes folder to be hub\music folder and made sure Create file with trackname was disabled as this was causing duplicate files all the time
• I added the cleanup folder to iTunes
• I allowed iTunes to consolidate the library which organized the files which renamed them and placed in the hub folder
• Now I had a lot of duplicate tracks, so I followed this advice to remove those:
• http://www.tech-recipes.com/rx/1579/how_to_remove_duplicate_files_from_your_itunes_media_library_

For some reason no matter what I do iTunes duplicates the files 2 to 4 times. As near as I can tell, it is doing this because it thinks the song could belong to multiple albums for that artist....Whatever the case, I am left with iTunes havoc and I am still puzzling out how to remove duplication in this folder without deleting original songs.

Thanks iTunes! Maybe it's not worth the bother.

Cleaning Up Photos and Videos

• I did a duplicate file clean on the photos and videos folder
• This is somewhat harder than managing music - as most cameras will tag a file with the same broad tag and then you copy 100 pictures labeled [Tag] 001, [Tag] 002, etc...
• It required lots of manual passes, and I made sure not to delete things I wasn't sure about
• I did some targeted searches using Windows Desktop search tool - looking for keywords and trying to merge things into broader folders
• In the end, I've gone from 2100 files (including 1102 duplicates) and 36 gigs of content to 998 files and 28.7 gigs of space.

Metadata is much more important for photos than for music files, because I can use them in many more ways (image editing, personal scrapbooks, social networking sites, websites).

Because an image can be used in many different places, it isn't easy to choose a single folder for a photo to live in. I've started experimenting with bulk insert into Picassa or similar so I can tag 'em. Also pondering adding del.icio.us to the Information Management plan since I am in the whole Cloud frame of mind.

How do you manage photos on your desktop, especially with metadata? Are there photo management tools you couldn't live without?

SiSense's Amazon S3 Analytics Tool: Update

Shortly after I posted about SiSense's new Prism Business Intelligence Viewer with Amazon S3 analytics add-on, their Product Manager Adi Azaria contacted me. I'd mentioned a bug that occurred when I was trying to view my files. He asked to see my Prism log files, which I sent, and provided me with the newest version of the product.

This version provides an enhanced UI including more help links, a set of "Common tasks", and the ability to quickly create new data connections. Most satisfying, it fixed my bug!

Looks like my bucket did not have logging turned on. The new version checks for this and prompts the user when this happens.

Thanks Adi for the quick follow-ups and the great support!

Amazon Simple Storage Service Analytics Add-On

ReadWriteWeb has a post on an Amazon Simple Storage Service S3 analytics add-on called Prism by the folks at SiSense.

It's in private beta right now but there is an invite code to the readers of the ReadWriteWeb blog. I've signed up to see how it can help me manage my Amazon S3 buckets.

Prism Viewer is a desktop installation that provides a Business Intelligence interface to view the S3 usage stats. The Amazon Analytics Dashboard is an additional file (Prism file: .psm) that lives inside the Prism Viewer and reports on your S3 buckets.

Once you've entered your Amazon S3 connection information, you can start to create reports on its activity.

Unfortunately I can't synchronize my bucket's logs because an error occurs. It is a private beta but I'm disappointed I can't yet see my own statistics. For the moment here's a screenshot of their sample bucket to indicate what the tool will look like:

I'll try again later and see how it works. Since I plan to store more and more information on S3 I definitely need this kind of reporting.

Amazon is starting to build a great ecosystem on top of its utility cloud computing platform, and software such as Prism, JungleDisk, TagCow, and DreamTeam Suite is the proof.

Eating My Own Dog Food

After 10 years of rampant, unsupervised personal computer use I've copied and pasted myself into a corner. I no longer have much idea where the good copy of anything is.

I aim to fix that.

It seems like a daunting task. My short term efforts have been to keep buying new external hard drives and copying files onto that. Needless to say that's compounded the mess since I am now trying to figure out which is the Latest Version. So much for Single Source of Truth!

In many ways my problem is a perfect mirror to the sort of enterprise problems that SharePoint hopes to solve. It only makes sense that I  make SharePoint a part of the solution. It has the added benefit that I am at least practicing what I preach; I will be eating my own dog food.

I thought I would track my cleanup efforts and hopefully get some help from other people who have gone down this road before. Feedback on what you did / are doing / will do is very very welcome!

The Problem, By The Numbers

• 2 users
• 4 Laptops in various stages of obsolescence
• 2 100-gig external drives
• 1 200-gig external drive
• 20 gigs of music including many duplicate files
• 30 gigs of photos and videos, including some duplication
• 10 years of ad hoc file management legacy crud
• 2 personal email accounts
• 2 or more business email accounts
• 0 ability to share files collaboratively
• 0 backup strategy

The Goal

At the end of this process, I want to have achieved the following:

1. A Single Source of Truth for all files (cue Heavenly Choir)
2. Highly available, online web access to files I frequently use
3. Laptop access to all files
4. Synchronization of file changes between online / offline work
5. Phone access to personal email, contacts, calendars, and task information (and files if possible, although this is secondary)
6. Phone access to business email, contacts, calendars, and task information
7. A single personal email account and preferably only 1 business email account
8. High level of security and encryption for all files stored in the cloud
9. Weekly (or more frequent) incremental backups of all files
10. Monthly full backups of all files

So here's the new plan:

The Bits and Pieces

1. Get a hosted WSS 3 site for my personal site. On here: tasks, calendar, grocery list, personal (but not security sensitive) documents, licensing information. Parts of it will require a username and password to login.
2. Get a business hosted WSS 3 site as an extranet for clients and also to store non-sensitive frequently used work documents
3. Purchase Colligo Contributor Pro so I can sync my Personal and Business SharePoint sites and work offline while I travel.
4. Get hosted Exchange (about to start trialing DNAMail).
5. Purchase a smart phone with Windows Mobile so I can read these lists, contacts, calendars, and documents while on the go using Exchange Mobile Sync.
6. The smart phone will initially also sync with my 2 personal and 2 business email accounts. Over time I will roll these over into 1 personal email account and 1 business email account (+x other company business emails when I'm on contract).
7. Designate a "Hub" computer to store every "non-cloud" file I need. This includes all my songs and photos - basically anything normally residing on a personal computer. The hub computer will actually keeps the files on a large external drive so I can plug all this data into another machine as required (such as when sharing files with my girlfriend's computer).
8. Get an Amazon S3 storage bucket for my hub computer. I calculate I need up to 60 gigs of storage for my photos, files, and music which amounts to about $180 USD per year including transfer and storage costs.
9. Purchase JungleDisk Pro 2 for automatic behind-the-scenes file backup of my data. Again, I need 60 gigs which amounts to about per $18 USD / year. JungleDisk offers encrypted online web access to all stored files.
10. All web interfaces such as JungleDisk and WSS sites will be SSL encrypted as well as password protected.

Creating A Single Source Of Truth

This is the HARD part....plowing through the files making sure I don't have duplicates. iTunes has already alerted me to the fact that my Music folder contains up to 4 copies of the same song! Dagnabit.

I'm currently investigating utilities that search for duplicate files, such as DoubleKiller and Easy Duplicate Finder (thanks to Lifehacker: http://lifehacker.com/software/lifehacker-top-10/top-10-free-windows-file-wranglers-330037.php)

For the moment I have created a new folder, called "Hub", on the Hub computer's external drive. I am manually copying the folders over and cleaning them up as I go. I will leave Photos and Music to the end in order to cling to sanity for as long as possible.

From now on, all work gets done from the Hub folder. If I need to access something and it isn't in the Hub folder yet, I will use that as the excuse to clean it up and move it. This is the "Go Forward" strategy for content migration.

Backup Strategy

1. Weekly automatic incremental backups to S3 using JungleDisk Pro.
2. Weekly manual site collection backup of the personal and business sites using SharePoint Designer.
3. Monthly manual full copy of the content on the external file drive to another USB drive that I store offsite the rest of the time.

Thoughts, etc.

Thoughts? Feedback? Sympathetic handshakes? Mocking laughter?

echo for SharePoint 2007 - Weekly Webinars

Every week, as part of my work with echoTechnology, I deliver a one hour online demonstration of some of echo for SharePoint's capabilities. It's very high-level and accessible and revolves around creating some tasks and showing how SharePoint can be updated easily and frequently.

The tasks I currently demonstrate include: loading content from a SharePoint 2003 portal and a file share, applying content type metadata, changing the master page and theme, adding some custom web parts, and enabling a custom workflow. All of these changes are deployed to several sites in two different SharePoint farms at the same time.

From the invite:

At the online demonstration you will discover how echo for SharePoint 2007:

• Easily replicates changes to SharePoint in bulk
• Automates change management procedures
• Tests and deploys changes portal wide
• Replicates to remote portals
• Rapidly loads and tags documents
• Delivers cost effective maintenance of SharePoint
• Provides a longer lasting more relevant portal for your business

The webinars are every Thursday at 1pm PST. Sign up online at the echo website: http://www.echotechnology.com/registration/Events.aspx?PID=8&EVTID=1&TimeZone=240

If you're a SharePoint Administrator you should definitely check it out. Hope to see you there!

TechEd 2008 Session Notes: MOSS Governance: From Chaos to Success in Ten Steps

This is the final set of notes from Tech Ed Orlando 2008. This presentation was given by Joel Oleson and Shane Young (and am I missing a presenter?).

Definition of Governance

Governance uses people, process ,technology, and policies to define a service, resolve ambiguity, and mitigate conflict within an organization.

Or:

People, Process, Technology, Policy!

The 10 Step Plan

1) Executive Sponsor

2) Governance Plan: Clear Goals, Vision, Mission, and Metrics: Reduce TCO, encourage standards & consistency, provide service and solution.

3) Skilled Teams:

Tip: Try to avoid having Jack of All Trades manage the portal. Even 2 half positions is better than 1 FTR (full time resource).

SharePoint specialist skills:

1. SharePoint Business Analyst – high level of understanding of SharePoint.
2. Creative Designer – often outsourced.
3. Trainer: Often outsourced.
4. Infrastructure Specialist – DB, OS, understands web world. Needs to know SQL issues and optimization of DB. Network understanding helps.
5. SharePoint Developer – knows .NET. SharePoint Architect – has to compile roadmap for services & platform, must be willing to follow the KISS principle.

Forrester organization defines these roles:

1. Executive Sponsor – CXO Evangelist, active participant in the process.
2. Program Driver – Communicator, unbiased, coordinator of all stewards.
3. Business Stewards – IT savvy, strong communicators, educator across the business.
4. IT Stewards – business savvy, educators across IT.

4) Training and Resources:

Site collection admin training – champs, pros, gurus (1 day).

Train key IT pros, dev, and designers – up to 1 week + offer certification incentive.

TIP: Brown bag and informal internal training for end users to help learn business processes.

Service offering site – should have online resources.

End User Training Kit for SharePoint should be available to everybody.

5) Service Definition and Model: Tiered model for site collections, sites, web apps, and farms. Options for things like quotas, user self-provisioning, archiving, listing in site directories, help disk provisioning etc, expiration of documents.

6) Information Architecture

7) Standards & Policies

8) Change, Configuration, and Release Management Processes (ITIL/ MOF) . Policies for release management, configuration management, operations monitoring, security and patch management

9) Culture and adoption. Enterprise search = easy win. Gravity: people gravitate to where their data resides. Don’t cut corners.

10) KISS

Examples of How Deployments Fail

• Cutting corners – “my devs are my admins!”
• “Production is test and dev, and where I introduce my service packs!”
• “SharePoint is a simple install!”
• “Let’s just deploy it and then worry about governance!”

TechEd 2008 Session Notes: Creating Portals That Last: Microsoft SharePoint Products and Technologies, Governance and Information Architecture

This is the penultimate in my series of Tech Ed Orlando session notes. It covers a session on governance practices given by Shane Young and Jason Medero.

Goals

1. Prevent content sprawl
2. Drive user adoption
3. Prevent single point of failure
4. Drive efficiency
5. User empowerment

Key Drivers

1. Information Architecture
2. Branding
3. Communication Strategy
4. Enterprise Search
5. Training / UAT
6. Operations Management

Things To Consider

Governance always has to involve the end users.

Use search reporting to decide what content to add / manage. It will illustrate what is being searched for and what can’t be found

Operations Management:

• Monitoring
• Backup/Restore / DR
• Storage and Quotas
• Service Level Agreements
• Reporting
• Deployment Process

Branding: Consistent Look and feel

Communication Strategy: Who, What, Where, When, How

Change Management

Users are like water – they follow the path of least resistance

Dev, Testing, Staging, Prod

Shane says 95% of customers do everything in Production

MOSS allows checking in pages and content

Shane advises customers to use the Dev, Testing, Lifecycle only for custom code

Need a plan for how changes are made

He never lets any custom code into the portal unless it is deployed via a solution

Where do I Start?

• Define roles and responsibilities
• Best Practice: Define success – make it quantifiable and tied to business objectives – usage rate, new functionality rollouts?
• Identify executive sponsors / key stakeholders
• Define taxonomy
• Develop look and feel
• Establish configuration and release management process
• Site provisioning
• Build a service offering
• At what level do I enable self service?
• Where do I want IT to step in?
• What quota should I allow?
• Offering types:
• Unmanaged: Out of box, hard to find, inconsistent experience, difficult to manage & patch
• Managed Decentralized: Easy to access, easier to isolate, distributed admin, challenges: brand, search, browse, support, backup
• Managed Centralized: Easy to index, manage, find information, brand
• Training / UAT
• Put training in the budget
• Quick reference guides for users

What to Expect from an Effective Governance Plan

• Protection against regulatory issues
• Enhance ability to find people and data
• Improve efficiency of your organization

TechEd 2008 Session Notes: MOSS Administrative Architecture, Deployment, and Operations (Part 2 of 2)

This was the second in a two-part series given by Joel Oleson, Shane Young, and Mike Watson. The session was highly technical in nature and consisted of some very valuable and very specific do's and don'ts for advanced SharePoint administration.

Tip: Add command line environment variable shortcut pointing to the Office 12 Hive location on your server, since you will have to go there a lot

Site Collection & Content DB Management

Tip: Put similar sites together (based on content sizes and # of users). This can help manage functionality and configurations.

Best Practice: Separate My Sites from portals so they can be moved and managed separately.

Best Practice: Large site collections should be dedicated to a single database.

Tip: Application pools:

1. 32 bit – can handle up to 1300 mb per app pool before recycle
2. 64 bit - up to 6 gigs per app pool

No best practice for how many Content Databases is the right number. This is more of a management issue for the business's IT staff.

Tip: Don’t put too much content into one site collection. Try to keep DB sizes under 100 gig – rough rule of thumb

Best Practice: Database Maintenance Plan: In SQL Server, do not use default growth plan, don’t autogrow in production, and plan sizes ahead of time

Disaster Recovery can be done via Stsadm, SQL Backups, Log shipping, remote snapshots

Best Practice: When Moving Config databases: use preparetomove, then detach and reattach (otherwise you get GUID problems!)

Customization and Development

You have to have a policy for how this happens. It HAS to use features and solutions or it doesn’t make it on the server!

Best Practice: Use Code Access Security (CAS) when coding

Best Practice: Test for memory performance and disposal of objects

Best Practice: Test adding and removing solution to the farm

Tip: To see which solutions are running – go to admin/solutionstatus.aspx

Best Practice: Deploy Solutions off peak hours, because all deployments will recycle the app pool which will dump user sessions. Also be present so you can test it!

Defaults Have Faults

Server:

1. Default log and app locations are not great
2. Basic setup is NOT recommended since it installs SQL Express
3. Having a different Application pool for every site collection is too much
4. No backup is scheduled by default

Site Collections:

1. No quotas, no expiration, no auditing
2. No versioning, no checkout
3. No approval or workflow required
4. SharePoint Designer allowed for all admins and designers

Database:

1. Max # of site collections is too high
2. DB file location and growth plan are wrong

TechEd 2008 Session Notes: Microsoft IT Best Practices

Microsoft IT ran several sessions on the topic of SharePoint governance. The ones I attended were Microsoft IT: Building Applications and Reusable Code on SharePoint, and Microsoft IT: How Microsoft Plans, Governs, and Operates Custom Applications and Portals within Microsoft. Both were hosted by David Johnson and Sean Squires. David and Sean did a great job sharing their practical experience and knowledge, and cultivating an open dialogue between themselves and the audience.

It was quite fascinating seeing how Microsoft "eats its own dogfood". However the resources they bring to bear make following their example a little problematic.  Microsoft IT use literally dozens of SharePoint farms, which from a licensing perspective alone would be prohibitive for any other organization.

SharePoint Structure

MS has several MOSS environments and many farms:

Vanilla SharePoint farms: staff can collaborate and use SharePoint Designer to tweak, but no custom code. Also includes My Sites. Most people use this.

Dedicated farms: Custom code deployed, no ability to use SharePoint Designer, supported by SLAs. They are currently trialing Enterprise Agreement-style process for helping Microsoft business units renew their portal use. They have to adhere to these agreed-upon standards or they can't use a dedicated farm.

Platinum – 7 major portals, specific customization and LOB applications, plus much more governance.

How They Do It

Created a diagram of information landscape and "pillars" - a catalog of existing code which allows reuse across the organization

Solution Catalog – list of custom solutions and samples

TIP: Don’t use MOSS as a relational database!

Best Practice: Standard application architecture - design for reusable components and in layers

Best Practice: All external UI labels and messages are stored in resource files (.resx) and SharePoint lists - this allows rapid UI changes without recompiling / redeploying. Configuration done via lists

TIP: XSL is critical to parse and present XML. Related to this, any custom UI uses token-based HTML templates

TIP: Make heavy use of Content Query webpart. There is a newer version of this with some changes to make it easier to modify

Security – typically use AD groups rather than SharePoint group. This is consistent with Microsoft's existing AD groups. Also when using SharePoint Groups, adding and removing users automatically causes a full recrawl. MSIT do create separate AD groups for publishers and creators

Runaway Successes

Vanilla SharePoint is basically a utility environment – employees can run wild subject only to quotas

MSIT – didn’t want to over-govern due to their highly technical staff

However this sometimes leads to “Runaway success”/runaway portals

Cleaning up a runaway portal becomes a matter of business negotiation

They are building new reports to at least assess what’s going on

To fix portal sprawl they are trying to migrate them back to vanilla environments, using scripts to do cleanup.

SharePoint Designer cleanup is much harder

Best Practice: You should organize portal on employee needs rather than org chart which often changes

Dogfooding

Microsoft has to upgrade early in the development cycle of the next generation of SharePoint, so vanilla environments help with migration

Dogfooding – Microsoft IT adopt 3 “9”’s for availability (99.9%) and no more than that, because they adopt Service Packs and new versions early

Best Practices for SharePoint Governance

MSIT are still trying to arrive at MOSS best practices themselves – they feel they are still in their infancy in terms of governance

Biggest Governance and IA challenge is breaking down organizational boundaries

Create a central place for governance

Create a central area for task checklists, best practices, and criteria for development

Best Practice: Create a “Playbook” of solution / configuration options – so business groups can look at this before they engage. This helps align them to existing solutions and the way SharePoint works.

Create scenarios – during consulting phase – to try to prove value of out of box functionality.

TE2008 Session Notes: Planning and Implementing Global MOSS 2007 Deployments

The next session I attended was given by Doron Bar-Caspi, Mike Watson, and Joel Oleson. It gave a great overview of things that need to be considered when implementing SharePoint across a WAN. The session covered the various deployment options, suggested information architecture, and gave tips and best practices to handle LAN performance testing.

Geographically-Dispersed Portal Challenges

• Cost vs Usability
• Bandwidth can be bought but latency is bound to the laws of physics
• User experience when accessing information across the WAN

General Options

1. Central Model
2. Central with Regional Portals Model
3. Distributed Model

Global Scale Search

Search Options:

1. Index centrally for everything. Impacts the WAN due to indexing but this can be throttled on the index server.
2. Index regionally for regional users. Each region hosts its own SSP. Results in two crawls but can be optimized for search

Federated Search:

New option, part of Microsoft's new Search Engine (which uses the SharePoint Search engine).

MOSS can now get results from remote search servers

My Sites

Usually SharePoint services are centralized

TIP: Consider synchronizing user profiles using custom solutions

Distributed model

Local portals handle everything including their own searching

Legal requirements can force this due to restrictions on data storage and use

TIP: WAN accelerators can help improve perceived performance when accessing information from other portals

Optimizing Authentication

NTLM is faster for smaller sessions, but Kerberos is faster for longer sessions.

Kerberos packets are larger, NTLM is easier to support and troubleshoot.

Which Topology to use?

• Know your usage profile
• Know your network
• Test from remote offices

TIP: Be aware that most work is done in My Site or project sites.

Best Practice: Evaluate bandwidth and latency before choosing a global deployment model. Also look at network maps first to decide whether hub and spoke model already exists.

TIP: Bandwidth and latency data was added to the Capacity Planning tool BEFORE the SharePoint model was added; therefore the calculations may not be fully representative of real world scenarios

Best Practice: Use ping to test latencies – using scripts to test at various times to develop a realistic understanding of network performance.

Consider end to end perceived network performance. Don’t assume anything: issues can be browser, computer, router, or network related

TIP: Use Fiddler to help assess network activity from SharePoint.

TIP: If you use the Business Data Catalogue it can be problematic because it makes big calls to SQL – no batching currently

Codeplex has a data population tool and whitepapers on managing geographically-dispersed SharePoint farms.

TE2008 Session Notes: MOSS 2007 Lifecycle Management: Methodologies, Tools, Best Practices

I thought I would upload my raw notes from Tech Ed 2008 Orlando in case they're of use to anyone. Hope these make some sense...

The first session I caught was given by Luis Camara Manoel and Satish Mathew, who are Program Managers at Microsoft. They gave a demo of new reporting tools which help to manage and report on SharePoint usage in an organization. The question they posed was:

How can you govern SharePoint if you don’t know what is out there? Discovery is required.

The emphasis of this session was on the following utilities.

SharePoint Asset Inventory Tool

Agentless utility; crawls network and reports on deployment and topology inventory.

How it works – runs WMI call to see if IIS is running somewhere. Then sees what web apps IIS is running.

Therefore “agentless” and CPU intensive only on the client, not the server (because it makes SOAP calls as if it were one user requesting).

Download: Currently in beta.

Executive Overview: http://technet.microsoft.com/en-us/library/cc295797(TechNet.10).aspx

SharePoint Capacity Planner

Helps see impact of various device utilizations: CPU, Network, Disk IO.

Steps to use: Assess, Recommend ,Simulate, Report. Can export models to Visio and Excel for further work.

Download: http://www.microsoft.com/downloads/details.aspx?FamilyID=DBEE0227-D4F7-48F8-85F0-E71493B2FD87&displaylang=en

Executive Overview: http://technet.microsoft.com/en-us/library/bb961988(TechNet.10).aspx

SharePoint Cross Configurator

Solution Deployment Package to enable / disable features and configuration. Includes audit, expiration policies, content types, master page deployment and enablement. Source code available in CodePlex at http://www.codeplex.com/SPConfigurator.

Leverages stsadm and makecab. To use: provide app.config file to setup.bat.

The tool will run against existing site collections only; new site collections will inherit the default settings. Only works across site collections, not across farms

Download: http://www.codeplex.com/SPConfigurator

Executive Overview: http://blogs.msdn.com/sharepoint/archive/2008/02/06/standardization-with-cross-site-collection-sharepoint-configurator.aspx

SharePoint Monitoring Toolkit

WSS / MOSS/ SQL / IIS Health roll-up

Uses System Center Operations Manager.

Download: http://www.microsoft.com/downloads/details.aspx?FamilyID=E4600FD9-F53D-4DED-88BF-6BB1932794F9&displaylang=en

Executive Overview: http://technet.microsoft.com/en-us/library/bb975149(TechNet.10).aspx

Virtual SharePoint Conference

Last Wednesday I had a chance to participate in a Virtual Trade Show for several SharePoint vendors. This was a nifty little vehicle to demonstrate our products to people all over the world.

The event was organized and hosted by DynTek, a consulting firm, using Live Meeting. They handled all the marketing and arranged for a representative of each vendor to present for up to 15 minutes on the product's capabilities and the value it adds to SharePoint. At the end of each presentation a live poll allowed the attendees to vote on how much a particular tool had captured their interest and notify us if they wanted a follow up demonstration or evaluation.

I ran the echo for SharePoint presentation. The other three vendors were Nintex (presented by Mike Fitzmaurice), ScriptLogic (presenter: Mike Perrault), and BA-Insight (Mark Aschemeyer). As the DynTek invitation put it:

Enhance Workflow & Reporting
Nintex will discuss how you can build complex workflow processes quickly and easily using a web browser interface and provide unprecedented insight into workplace usage, trends and behaviors.
Enhance Permissions
ScriptLogic will discuss how administrators can now backup and restore permissions as well as control the security of SharePoint environments the exact same, clear, intuitive way they would manage the security file servers - all from the same, intuitive console.
BA-Insight Software will discuss how to extend the search capabilities of SharePoint to deliver the most comprehensive and flexible Information Access Platform available in the marketplace today.

Enhance Management
Echo Technology will discuss how to easily migrate and manage change on your SharePoint platform through its entire lifecycle.
Enhance Search

It was a real challenge sticking to the 15 minutes time spot. It meant showing a few slides and setting the scene for why our products added value to SharePoint, and then (if time allowed) running a little demo.

I had a batch of echo Tasks setup ahead of time, so although the time was tight I was able to present live on how we could:

• Deploy 3 custom web parts from one site to another;
• Apply a new Theme and Master Page;
• Migrate files from a file share, apply content type metadata, and insert into a document library;
• Migrate files from a SharePoint 2003 portal into the same document library, again applying metadata as I went;
• Activate a Nintex workflow feature to a site collection; and
• Deploy a Nintex custom approval workflow to a document library

All of these changes were done from a local Development portal to a remote Production portal to demonstrate how easy echo makes that.

Kudos to DynTek for the cool concept and especially to Linda Ford, Julie Trautwein, and Dave Chennault who fronted the event. Well over 100 companies attended, and feedback has been very positive. Everyone seemed to enjoy the virtual format, which is so convenient. Hopefully DynTek will invite us to the next one!