Microsoft Announces Search Server Express

Today Microsoft announced that it will be releasing a free search tool for enterprises based on SharePoint and SQL Server. Search Server Express for organizations can be installed on a standalone server and leverages Microsoft's SharePoint search engine.

The engine looks just like WSS and SharePoint Search. The administration console has many of the same options as the current MOSS Shared Service Provider search settings. These include content sources, best bets, and scopes. Alerts and RSS feeds are available on the results.

Some of the neat aspects of the new solution:

• It supports the OpenSearch standard. This will help integrate Microsoft's search crawl and integrate with a wide variety of applications from other vendors.
• Federated Security: Authentication against content sources can be done via NTLM, Basic authentication, Forms Based Authentication, Kerberos, and "cookies". Access control lists are applied to the indexes and security trimming is provided on queries.
• Continuous Propagation Indexing: No need to wait for the whole corpus to be crawled before searching can be performed; the index will now be incrementally updated and searches can be performed on that increment immediately.

The express version can only be installed on one server. Naturally there is an upgrade path to more expensive solutions for those organizations that require scalability.

In 2008 Microsoft will also release some free connectors for Documentum and FileNet.

Microsoft has adopted a similar approach in the past with the releases of Visual Studio 2005 Express and SQL Server Express. In this case Search Server Express will provide a useful, free solution to give customers a taste of some core platform functionality and get developers using it.

The Enterprise Search website is located at http://www.microsoft.com/enterprisesearch/. You can download the release candidate here.

SharePoint As a Development Platform Debate

There's a very heartfelt debate raging at CodeBetter.com over whether SharePoint is a good development platform. The breakdown seems to be between "developers" and "SPs" or "SharePoint People".

Jeffrey Palermo kicked the discussion off by discussing what he feels makes a good development platform and then contrasting those points with the current state of SharePoint 2007 development.

I can understand the concerns of those who don't feel it is an ideal development environment. One of their biggest issues is how to manage the artefacts that SharePoint generates, especially the settings that can be made via the browser but not easily replicated to a source repository for sharing and versioning.

Many of these changes only live in one of SharePoint databases and so are hard to track. It goes against a programmer's mindset when these things are intangible and hard to track and synchronize.

Microsoft and the community are releasing more tools that can help manage this and perhaps in Office 14 there will be some easy facility to export and import configuration settings as an XML file. This would make it simpler to do proper change management with SharePoint.

There is also the fact that SharePoint is good at glossing over a lot of very complex interactions. Running underneath the hood is a SQL Server database, web services, a complex object model, Windows Server interactions, office integration, ASP.NET and Windows Workflow Foundation, a series of providers...it's very complicated but SharePoint makes it look easy. So when anything goes wrong somewhere in the stack it can be hard to figure out. In this sense it's a victim of its own success at abstracting the lower level plumbing.

The best SharePoint solution results occur when the customization points are kept in line with the core things SharePoint does well, such as workflow, collaboration, search, content management and publishing, and document management.

Whenever possible I recommend limiting SharePoint customization to things that can be done via the browser or through modifying XML or XSL properties (for instance for Search Results web parts). This also helps to protect the solution from service packs and future versions of SharePoint as any customizations are likely to be minor and easily tracked (and upgraded).

When new requirements come in, it's helpful to have the entire team of SharePoint developers compare notes on what the platform allows and how it can be customized. It's noteworthy how many times a 30-minute requirements-gathering discussion can end up with a shared agreement on how the out-of-the-box SharePoint components can be combined to deliver something that at first blush seemed to require half a dozen custom web parts.

The fact that so much can be done with so little effort actually makes SharePoint a killer development platform - the trick is to use a bit of imagination to align as much as possible to what SharePoint provides and go with the SharePoint flow. If SharePoint development is difficult and painful it's probably time to reassess the proposed solution to see if there's a more elegant, SharePoint-centric approach.

Anyway, there's a lot of fascinating insight being posted on Jeffrey's blog and anyone who hopes to develop on the SharePoint platform should at least give it a look. The full discussion thread is here. 

Don't Panic! The MOSS Market Opportunity

Microsoft Office SharePoint Server 2007 was released nearly a year ago now, and over the last few months I've started to hear people in the industry say things like "there is a 6-month window of opportunity for us to become leaders in SharePoint adoption!" or "the SharePoint wave is starting to crest and we need to ride it!". I find this attitude a bit puzzling.

It might arise because MOSS 2007 looks like any other packaged software product: you can double-click the .exe file and point and click your way to a portal that seems powerful and flexible enough to keep your end users happy. If someone thinks of it as just a product, it's easy for them to imagine that their competitors are scouring all the finite nooks and crannies of its feature-set and will corner the market "mindshare" on SharePoint work if they don't get there first. Hence the perceived urgency of early innovation.

However the real power of SharePoint is not as a packaged product, but as a platform. SharePoint-as-a-platform is the reason there won't be a cresting wave of adoption, but rather a rising tide that will lift all boats.

What makes the SharePoint platform most interesting, apart from its infinite scaling potential, is that the problems it addresses are enterprise problems. By this I mean the solution focus is on "30,000-ft level" concerns such as knowledge sharing, federated search, document and records management, team collaboration, and Line-of-Business application integration.

If you need a website, blog, or application built, then a thousand different software products or technologies could fit the bill. If you want any or all of those things plus a technology stack that abstracts away most low-level "plumbing issues" and allows you to address your enterprise problems, then the list of software you can standardize on becomes very small indeed.

The SharePoint platform is huge, and is only becoming more so, as Microsoft and third parties build upon it as quickly as they can. As one example, take a look at any of Microsoft's product line roadmaps, and see how many will be rerouted from standalone applications into SharePoint integration points and modules within the next few years. Over the short- to medium-term the platform will only become bigger and more complex, and the value-add potential will grow correspondingly.

There are an infinite number of ways a contractor or company can add value to a SharePoint deployment. Just to name a few:

• Document and Records management solutions
• Capacity, Performance planning and Disaster Recovery planning
• Portal branding and Look-and-Feel work
• Custom web part or feature development
• Custom workflows and electronic forms
• High-end consulting around building and maintaining a portal
• Portal migration and maintenance processes and tools
• Training for end users, administrators and developers
• LOB application integration
• Business reporting tools

Any one of these solution areas presents vast market opportunities in its own right. Each is also challenging and complicated because delving into it requires an equal-parts blend of domain expertise and an understanding of complex technologies. The enterprise issues that are raised require deep thought and effort to grapple with. Of course there is no requirement to concentrate on just one area, and therefore by working across these areas there is an unlimited number of potential combinations of SharePoint services and products.

Because no individual or organization can provide all of the pieces for a successful deployment, everyone is forced to share opportunities to some extent and can even choose to integrate some of their competitor's offerings without risking their profits or market mindshare.

I therefore believe that one of the consequences of competing on a platform rather than on a product is the creation of a surprisingly cooperative ecosystem. An economist would call this a virtuous cycle.

This is why I'm not worried about artificial "first mover" timelines in the SharePoint arena. Instead I foresee a long, upward trend of community growth as more and more SharePoint adoptions occur and the tools and knowledge to manage them increase.

So if you're worried about missing the SharePoint ride, just follow the Hitchhiker's advice, and "Don't Panic!"

OneNote and SharePoint: Better Together!

OneNote 2007 is great for note-taking. My colleagues and I have been using it for the last little while and we love it. There's always something new to learn about it.

After many months of using it individually, we decided to start sharing our OneNote notebooks with each other. Here's the cool bit: the notebooks can back onto a WSS 3 or MOSS library.

If you create a new document library somewhere on your SharePoint portal, you can choose a OneNote Section as the document template rather than the default Word template (although any library document template will apparently work just fine).

Now if you go back into OneNote, you can choose to share a notebook and host it in the document library you just built. It's easy to do; just click "Share" on the top menu and then "Create Shared Notebook". Give it a title and pick a template (we use "Shared Notebook - Reference Materials" or "Shared Notebook - Group Project").

Then when it asks "Who will use this notebook", just click "Multiple people will share this notebook" and choose to share it on a server.You will be asked to confirm the location: paste the URL of your site into the path textbox and then you can browse to the OneNote document library. 

Once created, you'll find a new folder in the SharePoint document library with the title of the notebook. Inside the folder there are OneNote files, one for each section (OneNote pages and subpages aren't actually individual "files").

You can invite people to the site just by giving them the library URL. Clicking on the OneNote notebook file will download a cached copy of it and open it in their OneNote client on their desktop.

We had a lot of fun adding content and pages, and watching as the clients synchronized changes first to the master files in SharePoint, and then down to each others' desktops. It's a very seamless process.

OneNote is very effective at synchronizing changes, although it doesn't use Groove apparently (Groove does file-based synchronization while OneNote seems to do more in the nature of content merging). 

You can view the standard Document Management panel in OneNote by clicking "Tools" and then "Document Management", which will allow you to view the site membership, see what's been updated, and set alerts if you want to keep on top of changes.

In order to search the OneNote content in MOSS 2007, you'll have to install OneNote 2007 on the Index server to get the OneNote 2007 iFilter, and then add the .one filetype to the Shared Service Provider's search settings. Instructions on how to do that are here.

This OneNote-SharePoin t combo is like a team Knowledge Base on steroids: fully searchable, disconnected, synchronizing, shareable, and with all the rich SharePoint functionality of any other library. Yet another example of the power of client software leveraging SharePoint's rich repository services!

Application Virtualization In The Cloud

Right now there are a couple of big industry trends I'd like to talk about: virtualization, and services "in the Cloud".

Virtualization is basically a shell in which to run applications: the software thinks it's on physical hardware, but it's all a sham. The first wave of virtualization was for servers and has allowed a great deal of consolidation in data centres, but it offers exactly the same benefits to client desktops and even to individual applications.

A variety of desktop and application virtualization solutions have arisen recently. VMware offers its Virtual Desktop Infrastructure (VDI). Microsoft has VirtualPC for the server and desktop, is adding a hypervisor to its next OS (via Windows Server Virtualization), and has purchased Softricity in order to get a share of the application virtualization market. Altiris is up to version 2.1 of its popular solution. Citrix (the new owner of XenSource) and ClearCube are other big players in an increasingly lucrative and competitive market.

Broadly speaking, these offerings all operate in the same way - the virtualization software is installed on some collection of servers that are centrally managed and hosted by any organization that wants to push the bits to its workers' desktops. In this model, the current virtualization scenario is essentially an internal deployment.

Now for another recent trend: Software as a Service (SaaS) or "in the Cloud". Web Services began the trend as globally distributed methods. Service Oriented Architecture is a little harder to define (due to vocal online debate), but Wikipedia makes the interesting point that in SOA, "independent services can be accessed without knowledge of their underlying platform implementation".

Many of the applications using these cloud services are thin client systems - browser-based web applications - although there is nothing stopping rich clients from using such services. 

Mix 'Em Up

Many software vendors are interested in making web versions of their software: witness the efforts by Google and Microsoft to host office productivity software on the internet. 

Another example is Adobe: At the Web 2.0 summit recently their CEO suggested that within a decade all of Adobe's applications will be offered online.

Like many traditional software makers including Microsoft Corp., Adobe must fend off rivals delivering competing applications over the Web and it also needs to adopt a new business model after years of selling software in boxes.

Chizen expects professional customers of products like Acrobat document-sharing or Photoshop for editing images would opt to pay for subscriptions versus facing a steady stream of advertising to use tools critical to their jobs.

This could involve re-architecting their applications to be browser-based. But what if they decided to wrap Illustrator or Photoshop in a virtual application shell and push it down to an internet user's desktop?

What do you get if you combine Virtualization with Software as a Service?

What Would The Advantages Be?

There are a huge number of advantages to desktop and application virtualization:

• No more DLL Hell: A core benefit of virtualization is its ability to shield your computer from registry edits, DLL conflicts, orphaned files, unstable services...You can even run multiple versions of the same system if required. This can be key if you have to use legacy software while upgrading - often impossible on the same computer if, for example, the versions of your software require conflicting Java runtimes.
• Pay Per Use: Need to use a piece of software temporarily? Download it at will and only pay for the time you use it.
• Application suite bundling: Doing some development work? Need an Integrated Development Environment, source control system, graphical editing tools, and testing software? Download a package of those applications for a flat fee. When you no longer need them, they will automatically be retracted from your desktop.
• Roaming software profiles: wherever you go, you can download and run the software you want, without worrying about installations and headaches like hardware-based license keys.
• The Latest and Greatest Version of Everything: When you download it, it's the latest version - guaranteed. No need to check for updates. The company providing the solution saves a bundle on maintenance and support costs by being able to centrally update their software - so they can pad their profit or passes on the savings to you.
• Platform neutral and optimized environments: the virtualized software doesn't care what it's running on because the company providing it will ensure that it is installed with everything it needs - and perfectly optimized in its packaged environment.
• Leverage the power of the desktop: hosting all these applications in a virtualized package on the desktop takes advantage of the client desktop's CPU, RAM, and other resources rather than pushing the burden back onto a server, scaling the solution out much more effectively.

So that's the dream. What kind of elements need to be in place to do this?

Pervasive Internet Access

If you don't have the bandwidth to download the virtualized software, then this will be a non-starter.

Luckily, most of the virtualization solutions just download the deltas to a user's machine, rather than all the code. Softricity for example streams the 15-20% of an application that a user needs to start up, and sends the rest as required.

As the percentage of the world that has access to the internet increases, and as high-speed connectivity improves, the market for virtualized fat client applications will gain as well.

Internet-facing Virtualization Systems

Right now most virtualization solutions are geared for internal deployments. A core assumption is that the recipient of a virtualized application or desktop is one of the organization's employees or at least a contractor. Softricity for example uses Active Directory accounts to determine authentication and authorization.

In a model where virtual applications are pushed out to internet users, security would obviously be a big concern. The company providing the package couldn't make many guarantees about the user's intentions. The pioneers in this field will have to take some calculated risks.

Federated Identity

One of the bigger (and longest-standing) web issues is how to reliably identify a user online. Any internet-facing virtualization provider first has to authenticate a user requesting software, and next has to authorize them based on any number of arbitrary business rules including licensing and financial standing.

These steps are non-trivial when requests are being served from around the globe. Many organizations are trying to create a web federated identity platform. Microsoft is experimenting with CardSpace to "provide [a user's] digital identity to online services in a simple, secure and trusted way". The Liberty Alliance Project is a group trying to build open standards for the same purpose.

Net Metering and Micropayments

If pay-per-use or monthly billing becomes the norm, we can expect a detailed breakdown of exactly how much we use any particular application: details such as the amount of data storage we require, how much upload and download occurs, whether an app is used in peak times or off-peak - essentially any of the details you would expect to see from your ISP today.

This sort of metering will be especially important because a user will rightly expect to be able to download and run software from any number of computerized devices.

Micropayments aren't strictly necessary but they help support net metering by allowing "pay per use" in cases where the cost is too small to make credit payments or the usual fees would make the transaction too expensive.

Dark Horses

Amazon.com is already operating in this arena. I often mention Google and Microsoft in the same breath because they compete so directly, but I think Amazon is the dark horse of the next generation web-as-a-platform race: when it comes to SaaS, they really get it.

They offer file storage in the cloud using their Simple Storage Service (S3). Costs range from $0.15 / gig for a month's storage, to $0.10 per GB transfer in to S3. EC2, their Elastic Compute Cloud (a massive virtualized grid computing offering) provides the same kind of pricing. Currently they are beta-testing Amazon Flexible Payments Service - which in addition to being a brilliant distillation of the online payment process, supports micropayments. What a great set of offerings!

It's only a very small step to imagine Amazon hosting virtualized desktops or applications in the Cloud.

So much for all my conjecture. Are we going to see this Virtualized Cloud Software model in the near future? I'd welcome your feedback!

Public Facing WSS Sites + Silverlight

Last night was another interesting Sydney SharePoint User Group. There were two presentations, one on using Windows SharePoint Services as a public facing CMS site given by Jon Matthes, and another by Michael Kordahi on Silverlight (+ SharePoint!).

Jon gave a very interesting talk about using WSS to rewrite a legacy web application for The Scottish Banner, an online community newsletter for Scots abroad. His aim was to improve on the old PHP website, retain its general design, allow online feedback forms, and produce minimal impact on end users in a cost effective way. He also hoped it would be an opportunity to learn about WSS 3.

The way he approached the site migration was to replicate as much of the existing structure as he could, with the only exception being the addition of the WSS administration area. He created anonymous access to a list to allow people to make submissions to the site while not logged in. An additional feature was the use of image libraries to serve as a simple product catalog, which was a new feature for the site.

Best of all, none of this required code. This turned out to be a good thing, as Jon had no access to the server and the US-based ISP was difficult to reach due to the time zone differences. Unfortunately he was unable to use Themes so some options were unavailable to him while he was doing the redesign.

Jon suggested some take-away lessons from his experience. He pointed out how important it is to use a hosting provider who has knowledge around SharePoint (and preferably one in the same time zone!). Research on what WSS could do and not do out of the box was important, and he also recommended making end user training an important component of any moves to SharePoint.

---

Michael Kordahi (http://delicategeniusblog.com) from Microsoft then presented Silverlight for us. This is Microsoft's intended Flash killer.

Its selling point is that it's a cross-browser/platform implementation of .NET for "building and delivering the next generation of media and rich applications for the web". Michael explained that its value is intended to lie in its flexible programming model combined with its ability to create rich media experiences.

A key component is its use of XAML, which will be the next generation declarative XML syntax for separating presentation from code and is already used in Windows Presentation Foundation. The goal is to have designers do their creative work in tools such as Expression Blend, and then have developers take the XAML code and code to it in Visual Studio.

There were some pointed questions from members of the user group. Flash can do all of these things currently, but it seems that the case for Silverlight might appear more convincing over time. Version 1.1 allows .NET code to be written on the server, and no doubt future versions will add increased functionality and productivity options. Already in version 1.0, it is easy to create animations and manipulate design objects, and then code to the Silverlight canvas using JavaScript.

Some people also remained unconvinced that the developer/designer split would be solved by this technology - after all that was an intended goal of code-behind. I haven't worked directly with enough designers to assess this problem for myself so I'm reserving judgement.

What was particularly cool about this presentation were all the demos Michael showed us. Tafiti is a neat experiment in changing the search interface to amalgamate multiple search content types. Inkable Search Tip uses the Vista handwriting recognition technology to do freehand searches on Google. Top Banana allowed browser-based video editing. A Chess Game written by Scott Guthrie rounded out the demos.

You can view many of these showcase applications at http://www.silverlight.net. Most of them are proof-of-concepts and showcases of what Silverlight can do, rather than real-world examples. 

All in all, it seems to be a very cool technology and one that'll become a very big deal. Silverlight 1.1 is currently in Alpha release and a preview version of Expression Blend 2 is available to MSDN subscribers.  For more information, Scott Guthrie has some excellent posts on Silverlight at http://weblogs.asp.net/scottgu/archive/tags/Silverlight.

Microsoft Releases End User Training Kit For SharePoint

Microsoft's SharePoint team is releasing a SharePoint End User Training Kit. This will be a helpful offering to clients who may not have lined up full training plans, or for use in self-service scenarios.

All of this content has been written by Microsoft's Office Online User Assistance team, evidently after some real-world experience responding to customers' questions.

The SharePoint Learning Kit version of the Training Kit is SCORM 2004 compliant and allows an organization to brand and customize the content delivery experience. A follow-up standalone deployment version will be available for individual use.

It's quite easy to convince end users of the power and convenience of SharePoint, but they may still feel overwhelmed by the options once they are sitting in front of the browser. Therefore user support and training is a key part of SharePoint governance and this training kit will be a welcome addition in the toolbox.

The SCORM version of the kit can be downloaded from the new SharePoint "Get the 'Point" blog. Incidentally this blog is focused on the SharePoint end user and has some very helpful walkthroughs of cool ways people can use SharePoint. It features some guest bloggers, postings from the Microsoft team, and links and tips to get the most of out a portal.

October Sydney SharePoint Usergroup and Office DevCon

Next Tuesday will be this month's SharePoint Usergroup. The main presentation will be given by Microsoft's SilverLight Evangelist, Micheal Kordahi. He'll be demonstrating SilverLight and talk about how it can be incorporated in SharePoint portals. This should be really interesting as it can probably add a lot of pizzazz to any SharePoint site.

Another presentation will discuss how to use WSS sites for public websites.

On a related note, the SSUG now has its own dedicated website, at http://SharePointUsers.org.au/sydney. It is a SharePoint site that uses the slick-looking Community Kit.

Finally, Microsoft will be hosting Office DevCon at their Ryde campus on Saturday November 3 and Sunday November 4. There will be many speakers and sessions covering all aspects of Office 2007 and the intended audience is developers and power users. The conference is free and you can register at http://www.block.net.au/devcon/index.htm.

Bob Mixon, Drew Jones, and Bob Fox have joined forces to launch the SharePoint Beagle, a free community newsletter dedicated to SharePoint. As Bob describes it on his blog:

Each monthly issue will be hosting a wealth of information on SharePoint
administration, development and end user answers

They've already signed up thousands of users so there's clearly a keen interest. You can view the newsletter at http://www.sharepointbeagle.com/.

SharePoint + Services

Microsoft has a new Software + Services vision, so this can't be much of a surprise: they are now offering hosted SharePoint "in the cloud".

Currently it's limited to organizations with "more than 5,000 employees" (so if you only have 4,950 you should top up) and includes hosted Exchange, Communications, and SharePoint all in a bundle called the "Business Productivity Infrastructure Online Services Offer".

It's definitely a good idea. I've had some involvement trying to come up with a managed SharePoint offering and the complexities of SharePoint administration and development make it a hard thing to price and support.

Microsoft looks to be avoiding that by making it largely self service, thereby placing their hosting effort on reducing the infrastructure and licensing burden which is I think a fair call and well within their control. I imagine they can also leverage their Fantastic 40 templates and the like to give customers some customization options.

Although it doesn't seem to be part of the basic SharePoint offering, something the associated Energizer case study mentions was the use of SoftGrid's Softricity to provide virtualized applications. This could be helpful to quickly roll out Office 2007 client deployments without causing problems and would fit well with the hosting bundle which includes Exchange.

The new Office clients would maximize the utility of the MOSS functionality, and providing them via virtualization would do an end-run around the problem many large organizations face, which is providing a quick turnaround on new Standard Operating Environments.

For a large organization this could be the "one stop shop". It'll be very interesting to see whether Microsoft can make this more widely available. Full details are here.

Attack Of The Blob!

(Image taken from WikiPedia's entry on The Blob!)

Yesterday I had a frustrating day trying to modify a CSS file for a SharePoint look and feel task.

For some reason, no matter what I did, the CSS never refreshed in the browser. When viewed in SharePoint Designer it did show the updated version, but no amount of IIS resets, file deletion, or browser cache clearing prevented my web browsers (both Internet Explorer and Firefox) from rendering the old version.

Eventually I happened to look on the file system and notice a folder on the root called "blobcache". Aha! I was using a new Virtual Machine and unbeknownst to me Blob Caching had been enabled to improve performance.

Inside this blob cache folder I found my css file with a .cache file extension on the end of it. I tried to delete manually but it then started to show "File Not Found." errors.

The solution was to flush the blob cache. The option is on the site collection's Site Settings. Select "Site Collection Object Cache" and then click "Force this server to reset its disk based cache."

The BlobCache is enabled in the web.config file and looks like this:

<BlobCache location="C:\blobCache" path="\.(gifjpgpngcssjs)$" maxSize="10" enabled="true" />

I could just remove the css reference from the path but during development I think the better approach is to just to set enabled to false.

Interesting note: David Marsh and I think SharePoint Designer was able to view the newly deployed file because it was using WebDav and so was hitting the database rather than having IIS redirect it to the BLOB cache which is what happened to the browsers.

Bottlenecks and Headaches

A new article on Computer World suggests that IT is a "key barrier to corporate Web 2.0 adoption", according to speakers at the Office 2.0 conference in San Francisco.

An example of how the IT team were slow to facilitate Web.20 projects:

[The] head of digital media at the National Endowment for Science, Technology and the Arts (NESTA)  in London, noted that she first had to bypass IT to get Web 2.0 technologies to the group's end users. Once she started the effort, she said, "IT started to realize it was happening without them anyway. They weren't interested until they started to get multiple requests from around the business. Eventually, they came on board."

The Pendulum Swings

Information Technology trends swing along a pendulum from complete decentralization to centralization. With Web 2.0 the pendulum is currently swinging away from central IT control.

In general I think this is a good trend,  since the pool of "developers" will deepen if it starts to include technical end users who are able to  create and configure their own systems (perhaps by mashing up a bunch of third party web services) or install and manage third party tools.

I'm just worried that a whole bunch of enterprise problems are being reintroduced by this decentralization, with much less transparent "ownership" of the challenges than if an IT team is in charge of development.

For instance, later in the article, the head of digital media at NESTA says that her organization

 uses many free tools, so it's easy to prove their value to management. "The value is instantly visible, especially when you do an ROI and your investment is zero," she said.

If a tool is "free", is it really zero investment? What is the opportunity cost of adopting and trying to integrate a free tool, without evaluating the alternatives? What are the costs of moving data in and out of it over time?

On page 2 of the article, there is this quote:

[GlaxoSmithKline Social media champion Lee] White said one of the biggest challenges to corporate Web 2.0 use is that demand generally bubbles up from users, while in the traditional corporate paradigm, upper management decides what tools workers should use.

I would suggest  that "demand generally bubbles up from users" anyway, whether Web 2.0 existed or not. After all, IT staff don't roll out of bed on Monday morning and decide to build a Leave Tracking System - they do it because HR users are requesting it.

I would argue that any "resistance" IT is putting up is probably due to its awareness of the incredible complexity of software development.

Hidden Complexities

Enterprise development is incredibly labyrinthine. A programmer's skill requirements range from user psychology to writing to research and analysis to understanding the basics of contract and intellectual property law - and I haven't even mentioned technical skills.

Keeping on top of information technology issues is a full time job. If end users are empowered to whip together "applications" using the latest services and tools then tremendous productivity gains may result - but what will the implications be?

Some of the open questions that spring to mind when I think about this new development paradigm are:

• Who fixes bugs in an End-User Developed (EUD) application?
• Who is responsible for managing code changes to an EUD?
• In the world of the permanent beta, what happens when a third-party web service that an EUD system relies on stops working or is discontinued?
• What happens when the service the EUD system is relying on changes?
• Who trains the other users to use the EUD system? Who documents the system functionality and business rules?
• What are the legal issues of using Web 2.0 systems in a corporation? Who owns the data? Who is responsible for adhering to the licensing terms of any components?
• What are the security implications of EUD systems that use third party software or services without any central oversight? Who ensures that security is properly applied?
• How does a company know what EUD systems are in use if there is no central control?
• How do these systems talk to each other? Is it important that they do?
• How does data integration occur if a variety of EUD systems expose the same types of information, such as user account details?
• Does a company need development standards anymore? If so, how does it enforce them on end users?

Are we heading from a bottleneck to a headache?

SharePoint Accessibility

One of the most painstaking web programming tasks is achieving web accessibility. Although estimates vary, between 5 and 10% of web users have some sort of accessibility need.

There are several standards, and determining which ones to support depends largely on your business needs and location. The US Government for instance has Section 508 compliance while the Canadian Government has Common Look And Feel 2.0 Accessibility standards.

Generally all of these standards build upon the specifications of the W3C's Web Content Accessibility Guidelines, specifically the Priority 1 and 2 requirements.

Some examples of WCAG requirements are that all web pages must have valid HTML and CSS code, images must have alternative descriptive text, and there must be alternative methods of viewing data or accessing page functionality. Accessible browsers such as JAWS cannot interpret client scripts or view images in the way that a regular browser can, but they can still provide a spoken translation of the markup code.

What happens if you have a SharePoint site that needs to be accessible? Well, unfortunately SharePoint currently outputs a lot of non-compliant code. Luckily, there is a lot of interest in the community in finding the workarounds, and SharePoint is nothing if not flexible.

 

Accessibility Options Within SharePoint

There seem to be a collection of elements that can go into creating a web accessible SharePoint site.

To begin with, you might wish to start with the Microsoft Minimal Master Page and a blank page layout. This will give you a fighting chance generating XHTML code, which is the first vital step. Zac Smith blogs about how to do this in detail.

Another requirement is to change the markup that the SharePoint web parts and user controls generate.

The easiest way to do this is to use a native SharePoint feature called "More Accessible Mode" that you can enable. This changes the UI rendering by applying Microsoft Active Accessibility (MSAA) standards for assistive devices. From the Microsoft Office Online site:

You turn More Accessible Mode on or off by pressing the TAB key immediately after placing focus on the page in a browser. Press the TAB key until you reach the Turn on more accessible mode or Turn off more accessible mode link. Even though the link to turn on More Accessible Mode is the first element in the tab order — which means that it is the first option that you interact with on the page — you may need to press the TAB key more than once to activate the feature.

Another way to change the rendering is to rewrite the output of web and user controls.

Web Parts are generally problematic as the HTML the web parts and the web part zones render is non-compliant. User Controls however are more easily changed. You could choose to do this using delegate controls.

At run time, the delegate control on the page selects the search control that has the lowest sequence number and displays the control to the user.

Another option is to use the .NET CSS Friendly Adapters that were introduced last year to help cleanup the markup the existing controls generate. John Ross (aka "Mossman") talks about this in his blog posting, "CSS Friendly Control Adapters in SharePoint 2007 (A Walk-Through)".

Client side scripts can be modified by using Regular Expressions to modify them at runtime (in the OnLoad methods of a page), but this may pose problems for future upgrades.

 

Third Party Solutions

HiSoftware has just announced that it will be partnering with Microsoft to create a Web Accessibility Kit for SharePoint:

The Accessibility Kit for Office SharePoint Server will include site templates, Web parts, documentation, instruction and tutorials enabling partners and customers to develop Web sites that conform to the guidelines set out in Section 508 of the U.S. Rehabilitation Act and the W3C’s Web Content Accessibility Guidelines (WCAG). The solution will also provide accessibility updates for Office SharePoint Server templates and Web parts for Web publishing, corporate intranet, team and project collaboration sites and more. This allows users to create a site and replace “out-of-the-box” MOSS components with accessibility kit templates, reusable Web content and Web parts.

Telerik offers a Rich Content Editor for SharePoint that contains more accessibility options and may be worth investigating.

I've heard SiteCore recommended as a good front-end for MOSS and it could be a useful alternative to a MOSS internet portal if meeting accessibility standards is a core requirement.

Third party web part vendors may generate more compliant markup.

Finally, you can follow the Glu Mobile approach and use SharePoint as a platform to help with content management and other functionality, but build an ASP.NET application on top of it, which will give you full control over what markup is generated.

 

More Information

• W3C Web Accessibility Initiative
• thekid.me.uk blog: Another Day, Another Accessible MOSS Site
• CSS Friendly Adapters
• SharePoint Accessibility - Is MOSS 2007 accessible?
• SharePoint 2007.com on Accessibility

If you have any tips or advice on how MOSS can be made more accessible, or any real world examples of MOSS portals that meet compliance standards, please feel free to post!

Hotfix City

I was cleaning up a portal today and ran into a wide variety of errors. Several of them turned out to have the same root cause. The symptoms were wildly varying event errors like this:

Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob threw an exception:

"Message: Old format or invalid type library"

Or

Microsoft.Office.Server.Administration.ApplicationServerAdministrationServiceJob threw an exception:

"Message: Not Enough Storage"

The resolution required not one but two hotfixes, both of which target the .NET 2.0 framework.

I applied the first hotfix, KB 923028, after reading about it on Gayan Peiris' blog. After rebooting I started getting another error which read:

Unable to open shim database version registry key - v2.0.50727.00000

Err....what?  Speculating wildly here (something I'm good at), I'm guessing the "shim database" was holding some key in the windows registry that the Timer job wanted to know about but couldn't get at.

Anyhow, I did a quick search to figure out how to resolve it, and there again Gayan's blog popped up, suggesting I download and run KB 918642. I figure he was two steps ahead of me the whole way!

By the way, Gayan's the new SharePoint MVP at Microsoft in Sydney. Check out his blog here. It's got a huge amount of SharePoint fixes. Hopefully you won't need any of them!

LINQ To SQL ≠ N-Tier Architecture?

Ok, LINQ seems pretty wonderful, and I can see the utility of it - and with regards to SharePoint we'll probably end up using it when Microsoft comes out with an official LINQ to SharePoint, or the community does. Heck, it may even replace (or at least abstract) the dreaded CAML code....

With LINQ, you get:

• Design-time type checking of data queries
• Code syntax that is consistent across objects, XML, and relational databases (and, really, anything else)
• Queries are integrated right into the .NET code frameworks
• Full IntelliSense support when coding what used to be a bunch of string values

All good stuff. But what I don't understand is how it fits with traditional N-Tier architecture. Is this more of the same old "you can write 70% less code if you don't mind your web page directly accessing your database" trend, or can we actually use this in a properly architected application?

I've been following along on Scott Guthrie's blog; he has a detailed series of posts about using LINQ, but he hasn't yet addressed where LINQ might fit in the N-Tier architecture.

Rick Strahl is already blogging about some of the limitations when trying to combine LINQ in the middle tier with POCO (Plain Old C# Objects), but it isn't clear if this is a beta 2 problem or a design limitation. Here's his conclusion:

The more I look at LINQ the more I'm coming to the conclusion that using LINQ in a middle tier - especially in a generic business object architecture - is not going to work well. There are many little problem issues that when all added up point at more problems being created than solved by the entity generation and easy CRUD layer.

Manning Publishing's upcoming LINQ In Action book has an early access program that lets you download PDFs and view chapters as the authors write them (the ink stains on the PDF are still wet!). It's a great read. Incidentally they're the folks behind LINQ to Amazon, which was one of the earliest LINQ extensions that showed up on the 'net.

Chapter 14 of this book talks about how to integrate LINQ into applications. It makes the point that LINQ "out of the box" is more of a Rapid Application Development tool - which goes back to the "70% less code" approach Microsoft is so fond of. However, the book suggests specific concerns that should be noted when trying to use LINQ in a DAL in the traditional way.

LINQ defers calls to the data source until the query is enumerated over, so if you're returning entities or queries to a business layer (in the format IQueryable<Whatever>, you might be introducing a dependency (your business layer is actually calling a LINQ to SQL query at runtime). You can turn off lazy loading by setting the DataContext's DeferredLoadingEnabled property to false, and also by explicitly loading an entity's child records before returning the entity from the DAL.

So, if I understand correctly:

• LINQ to XML or XSD doesn't seem to foil N-Tier architecture, since LINQ is essentially a tactical solution to make it easier to read or write XML, and doesn't cross any layer boundaries.
• LINQ to Objects is an in-memory LINQ technique, that again shouldn't cross any boundaries. Your method signatures will consume and provide POCO, and you use LINQ to iterate over them completely within the Presentation or Business layer. So far, so good.
• The real issue seems to be LINQ to SQL...and issues can be avoided if the LINQ to SQL code is confined entirely to the DAL, lazy loading is turned off, and disconnected entities and lists are passed back out of the DAL to the Business tier. But if you do this, the only apparent advantages of LINQ to SQL over any other ADO.NET data access method is the consistency of coding and the design-time safety of the strongly typed queries (which you can already get with Strongly Typed Datasets!)

Please, can anybody help me understand what an appropriate N-Tier architecture might look like with LINQ To SQL?

News Bits

Various bits of news and rumours:

• K2 blackpearl has gone RTM. You can download it from the K2 website after logging in with your user ID. Chris O'Connor is already blogging about the installation...check out his blog postings here and here.
• There is a SharePoint Asset Management Toolkit currently in beta at Microsoft. The intention behind it is to give an organization an overview of all the SharePoint assets floating around on the network: WSS sites, portals, etc. The idea being to get more of a handle on what's being built out there by enthusiastic 'Pointers.
• If you're keen on exploring LINQ, you'll probably be interested in the LINQ to SharePoint community project. Led by Bart De Smet, C# MVP, the project is currently in Alpha 0.2.3 (which is only four light years away), and the aim is to provide some of these following features (taken from Dimitri Clement):  

  • Custom query provider that translates LINQ queries to CAML, the Collaborative Application Markup Language used by SharePoint for querying.

  • Support for LINQ in C# 3.0 and Visual Basic 9.0.
  • Entity creation tool SpMetal to export SharePoint list definitions to entity classes used for querying.
  • Visual Studio 2008 integration for entity creation (a.k.a. SPML).
  • Can connect to a SharePoint site either using the SharePoint object model or via the SharePoint web services.
  • Planned support for updating through entity types.

  You can find the bits at LINQtoSharePoint on CodePlex and the team blog is at http://community.bartdesmet.net/blogs/LINQtoSharePoint/.

SharePoint Development Tips

Mark Jones has an excellent post on best practices for SharePoint development on his blog, entitled "Sharepoint 2007 - Development and Engineering Practices".

He places a great deal of emphasis on team development using Visual Studio and some sort of source control. He further recommends that as much as possible is automated via scripts or code to ensure that you can minimize the deployment effort, avoid errors, and manage code changes and versioning.

He's also a fan of test harnesses and not relying on SharePoint's object model to do all the testing for you.

One of my favourite tips is his advice to customize as a last resort. Although the SDKs and tools are getting better, there's a lot that can be done by configuring what's already out-of-the-box long before any code needs to be touched.  Office 14 is already underway and while I'm sure Microsoft will try to minimize the changes, it's a sure bet that enough will change between versions to make upgrading a roll-your-own portal painful. In fact it's amazing how much can be achieved with just a little thought about the existing features.

Mark's got a lot of great conclusions and ideas in there. I encourage you to check it out.

The Microsoft SharePoint Team has just announced a major release to the SDKs - version 1.2 of each. The full details are here: http://blogs.msdn.com/randalli/archive/2007/08/22/just-published-major-update-to-the-moss-and-wss-downloadable-sdks-8-22-2007.aspx

This is wonderful news - not only will it make developing to the object model more understandable but there are numerous samples and, best of all, a BDC editor tool! Check this page out:

The Business Data Catalog Definition Editor provides a visual tool for creating an Application Definition for BDC in MOSS 2007. Features include:

• Underlying XML is abstracted by the design surface and properties window
• Drag and drop web methods, tables, or views to create line of business (LOB) connections.
• Entities and methods are created automatically from database metadata and WSDLs.
• Additional method instances can be added to further enhance the database or web service connection.
• Method instances can be tested from within the tool, enabling incremental development of LOB connections

Background

Currently, writing an application definition to connect the BDC to a LOB system is a manual process. This requires an understanding of both how the LOB system is configured and what must be included in the XML to satisfy the BDC. Having a tool to simplify this process not only lowers the initial knowledge threshold for administering the BDC, it also lessens the required work of the user (such as testing, making modifications, etc.).

[...]

Highlights

• Tool supports databases (SQL, Oracle, OLEDB, and ODBC) and web services
• Drag and drop design surface for selecting DB tables or web methods:
• Metadata is automatically extracted from databases by dragging and dropping tables
• Web Services require a few additional steps to completely configure the connection
• Users can import and export Application Definition XML files
• Users are able to test method instances incrementally from within the tool
• The tool is not required to run on a web front-end
• Associations are created automatically when foreign keys are selected; they can also be created easily for web services by adding an Association method instance

Objectively speaking, the SDK and tool release just made life 100% easier (32.4% of the time)¹. You can't argue with those numbers!

The Microsoft team has also stated that it has doubled the resources working on the SDKs, which means we can look forward to even more improvements in the future. They're obviously listening to the community -they even make that point in their announcement - and it's very welcome indeed.

¹ Actual results may vary.

Yesterday the August meeting of the Sydney SharePoint User Group was held. The speakers were Han Duong and Brad Saide, both from LivePoint.

Han spoke about Themes in SharePoint. He gave a quick example of how to change a SharePoint site using the existing Themes in a MOSS installation, and suggested some tools, such as Heather Solomon's blog posting about all the SharePoint CSS classes, Firebug and web developer plugins for FireFox, and the IE Developer toolbar for Internet Explorer.

He then showed how to customize Themes using SharePoint Designer.

1. The first step is to copy an existing theme set from the Office 12 hive on the file system (<installation drive>\Program Files\Common Files\Microsoft Shared\web server extensions\12\Templates\Themes) and rename it to the name of the new theme you want to create ([Theme Name]).
2. Next, modify the name of the .info file contained within this new theme folder to be the [Theme Name].info. This is what SharePoint will look for when it tries to load your theme.
3. Next step is to tell it about the new theme: to do this go to 1033 folder in your hive and edit the SPThemes.XML file. Just copy one of the existing Theme element entries and rename it to reference your theme name.
4. Create a thumbnail for the them by copying an existing theme thumbnail from \Templates\Images and renaming it to whatever you specified in the SPThemes file.
5. Reset IIS
6. View the Themes in the Site Settings section of the SharePoint portal. You can apply your new custom theme to a site and then take a screenshot; you can then use this as the thumbnail to overwrite the one you copied.

An interesting point Han mentioned is that when you make changes to Theme css files in SharePoint Designer, it is making those css changes to the database. However, on initiation or cache expiry the the css and other theme information is loaded directly from the file system.

Therefore any changes made subsequently to the file system will overwrite the work you've done in SharePoint Designer. So he suggested that once you've made any changes you are happy with in SharePoint Designer, immediately copy them into the file system version of the css file to protect yourself.

Han mentioned that Themes are a good fit for simple design changes to the look of a SharePoint site, while Master Pages are more useful when changing the functionality of the site or the way it is organized.

Another good tip that was raised was using Themes as a quick way to mock up a client's site when doing demonstrations as it helps them relate to what they are seeing (rather than just viewing the out-of-the-box SharePoint installation).

---

Next, Brad gave us an overview of the Business Data Catalogue in MOSS 2007 Enterprise, and showed how it can be used to surface Line-of-Business information and provide dashboard views of disparate data.

He first explained the challenges of integration line-of-business code into SharePoint. These challenges include:

• Developers have to write integration code
• They communicate directly with the native API
• Each attempt is a single-purpose effort
• There is an ongoing maintenance and update burden
• It is difficult to create "one place to go" when there data is in many places
• Each new business system requires new effort

The BDC is a good solution for all of this. It provides a unified, consistent way to expose data within SharePoint by surfacing it from backend applications. It is declarative, requiring no code. It is also a centrally-managed system. I would also suggest it is "universal" or a contract that all developers will follow.

Brad pointed out the things the Business Data Catalogue is good for: surfacing LOB data, mashing up the information from multiple Line of Business apps, searching on all of this data at the same time, and pulling LOB data into libraries and lists where it can add to the existing portal information, and it can help populate user profile information.

What it isn't good for:

• It isn't a replacement for existing Line of Business functionality
• It isn't transactional or a message broker (a la BizTalk)
• It doesn't do data transformation (SQL SSIS)
• It isn't a data adapter (iWay)

Brad then presented a demo of how the BDC could surface HR system data stored in Oracle onto the MOSS portal. He showed how easy it was to use the free version of BDC Meta Man to build the initial BDC app schema file, and showed us how the BDC web parts could present and filter this information.

Updates to the Oracle database appeared upon refresh, and the BDC information could be added as metadata to existing lists and libraries. BDC Actions allowed HR staff to view the profile data of the records or even launch custom actions using URLS, such as launching a window to search on Seek.com for job candidates when a particular job title was selected on the SharePoint page.

I think the BDC is one of the best features of Office 2007 with some of the least documentation and fewest tools! A painful paradox.

Both Brad and Han did a good job and I think we all enjoyed the session.

If you're Sydneyside then you may be interested in attending two events this week: the .NET User Group and the SharePoint User Group.

The SharePoint User Group meets at 5:30pm on Tuesday at Unique World's office on Level 14, 24 Campbell Street.

Come along to the August User Group to hear Brad Saide from Live Point presenting on Business Data Catalogue and participate in a discussion with Han Duong from Live Point on customising SharePoint Themes.

In addition to Brad & Han's presentations, Jey from K2 will showcase K2 blackpearl, the BPM product from K2, and how it extends MOSS 2007.

The .NET User Group will be meeting at the Microsoft campus on Epping Road in North Ryde on Wednesday from 5:45pm to 9pm. This month Johann Kruse will be talking about Office Communicator 2007. You can find out more information about the event here.

I'll try to get more information on how exactly Office Communicator hooks into SharePoint (presumably through the existing "presence awareness" feature and there is also an API that can surface a lot of information).

Hope to see you there. Happy coding!

SharePoint Performance and Capacity Planning

This is based on Mike Fitzmaurice's seminar at Tech.Ed Gold Coast.

Mike defines capacity planning as the

"art of evaluating a technology against the needs of an organization and making educated decisions on how to meet those needs"

 He stresses that it is still more of an art than a science as there are a lot of imponderables and value judgements.

In terms of organizing the planning, he suggest breaking the process up into four distinct phases. These are:

1. Phase 1: Plan for software and hardware boundaries.
2. Phase 2: Estimate performance and capacity requirements
3. Phase 3: Plan hardware and storage requirements
4. Phase 4: Test, test, test your design

Phase 1: Plan for software and hardware boundaries

First you have to evaluate the environmental and the existing and possible limitations of the solution.

In SharePoint, hardware scalability isn't a huge concern as you can scale up and out by adding more servers over time. The big impact is in the software; specifically how you configure SharePoint and how you use it.

The main SharePoint objects to care about at this stage are: Sites, People, Search, Logical Architecture objects, and Physical objects (search indexes, physical database file and transaction logs).

These objects can all grow quite large, and SharePoint is designed to handle that to a degree, but there are finite limits. After running through multiple load-balancing tests, Microsoft has come up with the following benchmarks, beyond which performance suffers.

SharePoint Element	Recommended Limit	Negative Impact On
# Site Collections in single content db	10,000	General SharePoint performance
Site Collections	50,000 per web app	General SharePoint performance
Content Databases	100 per web app	General SharePoint performance
SharePoint Sites	250,000 per site collection	General SharePoint performance
Shared Service Providers	3 per farm TOTAL	Asynchronous processing, very seriously recommend you do not use more than 3 SPs per farm!
Indexed Documents (in Search catalog)	50,000,000 per search index	Indexing and querying performance
Web Servers	8 web servers per database server	General SharePoint performance
Maximum # documents in flat document library not using folders or indexed views	4,000	Searching, viewing, and navigating document library
Maximum # documents in document library using folders or indexed views	1,000,000	Library performance
Maximum # documents per nested folder in document library	2,000	Library Performance

 

Some other helpful tips:

• Don't create too many distinct web sites in IIS when you are installing SharePoint for the first time, as the application pools and websites use up CPU resources
• Create usage profiles to try to track common users and estimate how their daily actions might consume resources or generate content
• Mike really recommends that you turn on IIS compression.
• BLOB Caching is a feature that isn't well known in SharePoint but serializes large objects to disk on the Web Front Ends to avoid database round-tripping.
• You can delay loading the core.js file if your users will be anonymous or readonly...instructions are here.

Phase 2: Estimate performance and capacity requirements

The requirements are a compromise based on your desired performance, capabilities, and resources.

Remember to plan for peak concurrency since your farm could be humming along most of the time until peak demand brings it to a crawl.

Microsoft recommends 64bit hardware to improve performance, but be aware that many applications and utilities have problems with this: PDF Filter is one example (although there is now a 64 bit version). MOSS supports mixed mode where your database server is running on 64bit hardware while the Web Front Ends are on 32bit boxes.

Phase 3: Plan hardware and storage requirements

If you're going to be storing your file system in SharePoint, multiply its size by 1.2x to 1.5x to reflect the additional metadata, storage, and related costs to that. Your database will have to accommodate at least this much space.

Plan an additional 30% to be the size of the Index for all content indexed for a single server.

If you have a dedicated Query Server multiply the Index size by 2.5x.

If you're enabling BLOB caching in SharePoint, remember that it is serializing large objects to file, and you will have to have room for this.

Plan for future database growth!

Phase 4: Test, test, test your design

You should definitely test the performance under a variety of scenarios.

To do this, first establish goals for your tests. Good goals might be to mimic the standard user profiles you created in Phase 1 at peak times and see how the SharePoint farm responds.

Create a test farm that closely mimics the production farm. I think it's ok to use virtualization as it will cut down on the number of physical servers you need. Populate the test farm with data that is representative of the real thing. You could try exporting existing content databases from a development or production farm into the test farm. There is also a data population tool available on (where else?) CodePlex. 

You can use Perfmon counters to assess responsiveness. There is a tool called Fiddler that can help you analyze the requests and responses between SharePoint and the client which can help figure out what's going on.

SharePoint can be very complex because there are a lot of moving parts. One of the most important aspects of a proper MOSS deployment is planning for future capacity and performance, and these tips by Mike are a great help in that regard. The white paper that discusses many of these issues in greater depth is available here.

Tech.Ed 2007 Gold Coast Day 3

The third and final day of the conference was more of a wind-down, but still had tons to see and do. I caught the last bit of the first session on Rules Engine Use in Windows Workflow and then made my way to the seminar on Best Practices for Team Based Software Development. This was mostly about how Team Foundation Server can help organize the common artefacts and work habits of a development team.

The speaker, Anthony Borton, covered the signs of a great team and mistakes that teams can make as well. He then displayed a chart of various stages of the Team Development Maturity Model he had developed with a teammate of his. The stages were:

1. Chaos
2. Shared - standard location for team documents and not much else
3. Tracked - use of standard forms and code
4. Defined - improving level of automation for common tasks
5. Instrumented - stabilized metrics and repeatability
6. Optimized - continual process improvements, tool refactoring, ever-increasing automation

I think this is a fairly descriptive overview of the stages of collaboration a team can achieve, although I'm not sure how concrete or prescriptive it is.

Anthony also talked about the requirement to do code branching and best practices such as structured feature decomposition (which I guess means breaking down the business requirements into smaller units of fertilizer).

---

Next up, Mike Fitzmaurice talking about Capacity Planning for SharePoint. Lots of interesting information here, which I'll cover in more detail next week. The gist of it was that capacity management is an art, not a science, although there are plenty of metrics in it.

He described a way of dividing a capacity planning study into discrete phases so that issues could be dealt with in a systematic and holistic way.

The first phase is to plan for the software and hardware boundaries. The next is to estimate the performance and capacity requirements. Phase 3 is to plan hardware and storage requirements; and the final phase is to test the design.

Again, he had some great facts, figures, and tips which I'll cover in greater detail in a dedicated post later on.

---

The final seminar of the day was given in a packed room by self confessed "language geek" Joel Pobar. He's a former Microsoft Program Manager on the common language runtime (CLR) which underlines everything we do .NET-wise. 

This presentation was a virtuoso overview of computer languages features and trends...especially the combination Microsoft is trying to achieve between Static languages, Funtional languages, and Dynamic languages.

Hopping into PowerShell command windows, Joel created language demonstrations on the fly using a variety of languages interchangeably, including JavaScript, IronPython, and the next version of VB.

At one point, to a huge ovation, he created a LINQ to SQL statement that generated xml items in its select element, which were then added to an RSS XML fragment that hadn't even been declared yet, but would exist at runtime!!!!

Now I'd love to talk more about this, but it was so far over my head, I felt like the man-ape trying to figure out the Monolith (but hey, at least the ape figured out tools right after that!)....It was just one of those moments when you know something special is happening but you're not quite sure what it is. So I just went "woooo" when everyone else did, and nodded my head emphatically at various regular intervals :) 

Guess I've got some more lurnin' to do!

---

After this, all that was left was the wrap-up. The beginning of the end was a locknote speech by the head of the Computer Forensic team of the Australian Federal Police, and then it was time to say goodbye to Frank Arrigo (who is heading to Microsoft Corporate in America), and watch Alan Coulter win $20k worth of prize swag (no idea how he got it home!).

I'm looking over my (long!) blog postings of the three days of Tech Ed - it's amazing how much learning, networking, and fun can be crammed into such a short space of time. This was my first development conference and definitely not my last! Kudos to all the organizers for such a great job.

Tech.Ed 2007 Gold Coast Day 2

Day 2 was a full one. Software + Services and Web 2.0 in the Enterprise were two seminars given by Michael Platt but again I won't report on these since I'm going to blog about all the Web 2.0 stuff in detail later this week.

Enterprise Library 3.1 was given by Tom Hollander, who has left Patterns and Practices Group and moved back to Sydney. He stated the goals of 3.x were to:

• Address developer feedback;
• Provide new blocks to address standard issues, such as Validation and seperating policy from business rules;
• Support integration with the new features in the .NET 3.0 framework;
• Simplify development of new application blocks;
• Retain backwards compatibility with EntLib 2.0

He demonstrated the major new blocks, showing how easy it is to create validation rules programmatically, using attributes, or via configuration files. The validation block allows developers to specify the rules once and have them apply everywhere - in the presentation layer as well as throughout the other tiers.

The Policy Injection block is a neat idea, since it allows the separation of cross cutting concerns such as caching, logging, and validation - in short managing many of the other blocks in the Enterprise library.

One potential issue is that any code using the Policy Injection block has to inherit from MarshallByRef, which means if you have a custom framework inheriting from a custom base object, you may not be able to use this block.

He also demonstrated WCF integration and the use of Handlers to place Enterprise Library functionality in the IIS pipeline, which is very convenient as you can sit them in front of any application without writing any code.

---

The next seminar was again given by Tom and was another conference highlight for me. This one was on Software Factories, a topic I am very interested in and that we will be hearing much more about.

Tom explained the challenges faced in solution development by describing the "gap" which exists between the general purpose frameworks provided by vendors such as Microsoft, and the solutions developers build upon these frameworks. The gap incorporates things such as business knowledge, requirements, architectures, technology decisions, and so on, and represents the additional effort that has to be undertaken with each new project.

Development teams currently bridge this gap with skilled labour, but this costs a great deal, takes a long time, introduces risk to the project, provides relatively poor quality and consistency, poor traceability, etc etc. Basically we know that the current development process is difficult and crude!

The software factory attempts to close the gap as much as possible by combining software automation (code generation) with best practices and guidance (domain expertise). As he defined it:

It's a collection of tools and assets for building applications of a particular type. When turned on in a development environment, it provides a guided experience to the members of the team."

The software factory concept is related to Domain Specific Languages (DSL) which are really languages or code targeted for a specific domain, such as Banking or Medicine.

Tom demonstrated the use of the Patterns and Practices group factories that currently exist:

• Web Client
• Smart Client
• Mobile Client
• Web Service

Microsoft is hoping to bundle software factories into the next version of Visual Studio (after 2008). As an early demonstration of the "factory of the future" he showed us the "Modelling Edition" (the name makes it sound like a Fashion Industry DSL).

Using a UML- or class designer-style interface, he dragged and dropped units of functionality onto a canvas to create an abstract model of a solution, and was then able to generate the code from it. It's a compelling vision and where the industry needs to go to reach the modern ages of computer development, but we're still a long way off.

For more information, check out the service factory on CodePlex.

---

"Stranger in a Strange Land: The Adventures of a SOAP Fan" was a fascinating and often hilarious exploration of the web as a middleware platform, given by Dr. Jim Webber whose blog is World Wide Webber (no relation to the World Wide Web).

Dr. Jim discussed the major architectural characteristics of the web:

1. It is scalable
2. It is fault tolerant
3. It is recoverable
4. It is secure
5. It is loosely coupled

He made the valid point that these are precisely the virtues that we would seek in any major enterprise application. Having concluded that the web's nature makes it a desirable platform, he proposed that we should be building our systems using its fundamental nature.

He had several proposals:

1. Links are good. In fact, if the web itself is a state machine workflow (resources are "states" and requests are "transitions") then links represent state transitions.
2. HTTP verbs provide all the basic functionality an application needs: Create, Read, Update, and Delete actions, plus metadata provided by HEAD and OPTIONS verbs and Status Codes that assist processing by letting us know the result of our method calls.
3. Microformats will create the small "s" semantic web. Tim Berners-Lee's vision of the Semantic web will never be realized since it ignores the fundamental reality that the web is a chaotic human creation, but microformats are entirely achievable and can provide real context and knowledge.

He then gave a funny example of how purchasing a Starbucks coffee could be achieved using this model. I still think this development model is a stretch since the tools and development mentality don't really support this yet, but as a thought exercise his ideas were top-notch.

---

The final seminar was "Building a Complete Web Application in ASP.NET 3.5 and Visual Studio 2008 (Part 2)" given by Jeff King. He gave a great presentation, creating functionality on the fly. He spent a lot of time demonstrating the powerful new JavaScript support that's available. Although he did a fine job I still remember writing JavaScript code during the Browser Wars and in spite of the advances I think I will need to give it a few more years before the mental scars heal :)

---

After that it was time for Movie World. Although I only had time to ride in the Batman Drop-thingy and on the Superman roller coaster once both were great. Superman especially was crazily fast. After the park closed (pretty early, really) I headed out to Surfer's Paradise with a couple of Microsoft guys, Richard and Shannon. We ended up leaving when everything shut down at 4:30 and I crawled into bed at 5 for a couple of hours of shut-eye. It's a hard knock life for us programmers!

Tech.Ed 2007 Gold Coast Day 1

So Tech.Ed 2007 came and went on the sunny Gold Coast and over the next few days I'm going to be sleeping and blogging (not necessarily at the same time). Here's my recap of the first day:

The keynote speech was given by Michael Twigg, the Production Resource Manager of AnimalLogic, an Australian animation firm that makes special effects for movies, TV, and commercials. Some of their achievements include effects work for the Matrix (the ghost twins), 300, and Happy Feet. It was an interesting example of IT in an industry I'm not familiar with.

They can spend months designing a shot that will be used for two seconds. As one example Michael showed us how many levels of matte painting, 3d effects, and live shots that all had to be pasted together to create a two-second shot of Xerxes' army crossing into Greece for the movie 300.

We also learned the surprising fact that all their servers run Windows NT! Well, at least he didn't mention the "L" word (no, not that "L" Word - the other one).

---

The first seminar I attended was "Lap Around Visual Studio 2008" by Tony Goodhew, the Product Planner for the product. Tony described the major goals of the product version, namely language advances, web development improvements, and increased support for AJAX and JavaScript, and CSS.

He covered some of the neat new features in VS2008, including framework targeting, nested master pages, the CSS property windows, split pane view, and LINQ to SQL.

Other things he touched  on:

• Hitting CTRL while using IntelliSense makes it translucent so you can see the code that is below it.  Highlighting code while hitting CTRL-K-D formats it, which  is useful when you are pasting XML or HTML fragments into a page.
• Tony also talked about XAML and how Visual Studio is the developer's platform, while XAML presentation can be handled in the new Expression suite that Microsoft has released. XAML separates the concerns so both roles can work in parallel and neither overwrites the other's work.
• There is a service editor in Visual Studio for WCF projects. Not sure if this is included in the previous WCF extensions toolset but it looks quite useful.
• There is support for local data caching using the SCCE-based local data cache. This is intended to help provide a standard solution for applications that may have infrequent access to a network. Tony demonstrated how to setup the wizard driven cache and then showed an app connecting and disconnecting to prove that the local cache would synchronize properly.

---

"Project 2007 Timesheets and Reporting" was the next seminar on the list. It covered the various options for using Project 2007 and Project Server 2007 for creating timesheets, assigning and tracking human resources, and presenting and reporting on this information. 

As expected, the new version of Project Server 2007 uses WSS 3 project sites to host Project actions, plans, and reports.

When an employee logs in to the Project Timesheet site, he is presented with a series of links, grouped into "Tasks", "Timesheets", "Approvals", "Status Reports", and "Issues and Risks".

Each employee has a "My Timesheets" view, which seems to be a list using a custom content type. The content type allows him to create a new Timesheet entry, and the items in the entry can be divided up into project- and non project-related tasks (for administration, research and development, or other purposes). There is also the obvious notion of billable and non-billable hours.

Project Managers and Timesheet Managers get top-level approval and rollups of all the timesheets and project plans, as you would expect from such a tool.

Managers can also view predictions of future timesheet entries, for instance when an employee submits a request for leave. On the My Timesheets screen there is a handy little icon that shows whether a vacation request has been approved or not (it looks like a KPI).

---

Next I attended a seminar called "Web 2.0 Programming", by Microsoft's Director of Web 2.0, Michael Platt. I found this to be so insightful and it dovetails so well with a couple of other seminars that I am going to blog about those separately.

---

After the last seminar of the day (using Dynamics CRM 4 for handling events - basically hooking CRM up into your business platforms) I headed off to the K2 Underground Party, hosted at Onyx just down the road from the Conference Centre.

At the party I finally had a chance to meet some of the people whose names I have heard or blogs I enjoy reading. I met Jey Srikantha and Chris O'Connor, aka "Grumpy Wookie" (who turns out to be a fellow Ontarian by birth!) , and I also met Anthony Petro who I had previously interviewed and Chris who worked on programming K2 against the SharePoint object model (I'm sure he enjoyed it!).

Some of my colleagues from Dimension Data - Jeremy Hancock, Ben Johnston, and Alan Coulter - were also there, and we caught up with some former team members, amongst them K2 Insider Bruce Swiegers (who blogs at http://k2insider.blogspot.com) and Ian Newark, who is a Business Development Manager at K2 now.

It was a good group of people and lots of fun. Best of luck to K2 with their product; it seems like they are seeing a lot of interest from the dev community so far. I even saw one Tech.Ed attendee covered head to toe with K2 Underground stickers - I guess you could say he is committed (or should be)!

I'm lucky enough to be spending the rest of the week at Tech Ed 2007 Gold Coast, courtesy of my managers at DiData.

The seminars this year seem very interesting - there's a lot of coverage of Web 2.0 concepts. Microsoft is focusing heavily on what they call Software + Services (it's kind of like SaaS with some math, I guess) and there is heavy emphasis on this in the lineup.

In particular I'm interested in learning more about Enterprise Library 3.1 and Software Factories from Tom Hollander (formerly of the Microsoft Patterns and Practices Group). Among the scores of other sessions are ones on workflow, the latest version of Dynamics CRM, capacity management planning for SharePoint, and Business Intelligence.

There is also a Blogger's Lunch about Web 2.0 trends with some guest panelists and questions from the floor. Should be a lively discussion.

I'm looking forward to learning plenty of new things and meeting lots of my fellow super-caffeinated developers. I'll try to blog while I'm there.

Currently Team Foundation Server uses Windows SharePoint Services sites to host project sites. This provides a useful central point to collect development documentation, workspaces, meeting spaces, and allows people without TFS to at least view the related information (although now there is an additional option, the very cool TeamPlain Web Access for Team System, which is free for all licensed TFS users!).

Unfortunately the current version only supports WSS 2.0 sites. There are some work arounds, but they can be flaky.

The good news is that TFS 2008 will support WSS 3 sites - Abdelhamid Abdou talks about this on his blog in this posting. Better yet, when you setup TFS you are now given the choice to have the SharePoint integration on the same box, or point to an existing SharePoint application on another server.

To learn more about the current workaround, you can read Brian Keller's blog posting here. Brian is a Technical Evangelist for TFS and makes it clear that any modifications to current TFS installations to make them work with WSS 3 are hacks and may not be supported. He does point to Mike Glaser who blogged extensively about how to upgrade to WSS 3 sites.

Incidentally, as of July 30, Readify in Australia has released TFS hosting "in the cloud"! This could be very huge for small ISVs or consultants that want to have enterprise development tools without the upfront expense or maintenance effort! The service starts at $995 AUD a month and more info can be found at http://www.tfsnow.com/.