Posts tagged with "Siebel"

In my job as a roving IT consultant, I have given a number of technical presentations about Siebel. During that time, I have learned that I am much more comfortable presenting material and content that I have created myself. I have also presented slide-decks used by technical pre-sales. This has occasionally led to detailed questions arising about a very innocuous looking bullet point which I was unable to effectively answer; not a comfortable situation.

Occasionally, I have delivered technical workshops about a very specific area of the Siebel product set that was tailored to a customer requirement for a module that is not covered by a formal course offered by Oracle Education. I have thoroughly enjoyed this type of work as I find it stimulating and very rewarding. I also felt the customer also found these workshops useful and valuable.

Last week, I gave a 2 day workshop about EIM (Oracle's ETL tool to bulk load data into a Siebel OLTP database) and I actually created some practical lab exercises to give the attendees some hands-on experience of failing to load data using EIM. I'm being serious here - I firmly believe it is very important to be exposed to the different classes of errors and idiotic mistakes when trying to achieve a seemingly straightforward task of populating a single customer record.

I visited the client's offices the day before to check the logistics and the Siebel environment provided for the workshop. I also took the sensible precaution of completing the various exercises myself. To my horror, I discovered at 3pm on the day prior to the workshop that EIM wasn't actually functional in the customer test environment. Thankfully, I discovered this was due to a missing Service Pack for the Microsoft SQL Server database which resolved the problem and saved me from a very embarrassing situation.

Whenever I've attended training courses, I've always felt slightly uncomfortable whenever the instructor went walkabout and hovered over my shoulder as I struggled with the syntax of ‘ALTER TABLE'. Consequently, when I set the attendees loose on the first exercise, I tried to take a back seat and only help if someone requested assistance.

I am not a teacher - in fact, I am a lousy teacher as I have precious little patience - and I am not a trained instructor. However, I was quite curious to see how different people attacked the problem. One chap was feverishly reading the manuals, typing at speed, running the tests, examining log files, iterating in an effort to win the race. One of his colleagues had a rather more considered approach and I noticed he chose to take time to assist his neighbour who wasn't as far forward. Another attendee was very methodical and thorough; he essentially created a full source-target mapping in Excel before he did anything else and was completely unfazed by the progress of others around him. Another gentleman reminded me of myself - he was bludgeoning forward at breakneck speed, making mistakes (syntax errors), immediately fixing them and iterating rapidly.

I approached one gentleman quietly beavering away in the corner who had actually completed the original exercise but hadn't shouted ‘Eureka' or called me over to praise his efforts. Instead, he was now creating a data set of 10,000 customer records to see what throughput he could get compared with the performance of the existing custom COM based data loading tool.

All in all, a very enlightening exercise. Psychologists would probably classify each type of individual with a special name (‘Starter-Finisher').

Siebel 8.0 is the first major release since the Oracle takeover and is now imminent.

Oracle are planning a simultaneous launch of five different products (JD Edwards, E-Business, PeopleSoft and Siebel 8.0) on 31 January 2007.

Ed Abbo (VP CRM products) will be chairing the Siebel session and SearchCRM has an interesting (albeit low quality) interview podcast with Ed where he talks about new features in Siebel 8.0, CRM OnDemand and the impact of the Oracle takeover on Siebel's development and strategy.

The Oracle Applications Users Group (OAUG) also has an excellent summary of the new functionality in Siebel 8.0.

IBM has published a useful whitepaper about tuning Siebel 8.0 on AIX 5.3. The paper includes benchmarks with different virtual memory page sizes and shows the benefits of simultaneous multi-threading. The paper also includes analysis of generic tuning features available in Siebel (connection pooling and threads per process).

The two key features that specifically interest me are the Siebel Diagnostic Console and the introduction of support for Linux (RedHat 4.0, SuSE 9.0).

The Siebel Diagnostic Console offers an Analytics style dashboard interface into the volumes of data produced by SARM.

Secondly, the number of Siebel customers choosing to deploy on Linux in the coming months will be interesting.

Another example of the closer integration between Siebel and Oracle is the recent announcement of a management pack for Oracle Enterprise Manager (OEM) in the recently released 10.2.0.3.

Oracle Application Management Pack for Siebel is a comprehensive solution for managing the configuration, performance, availability, and service level of Siebel CRM applications. It can be used to monitor the health of the servers and components, measure application response time, track configuration changes, and diagnose performance and execution problems.

This will be an invaluable tool for Oracle DBA's (who may be unfamiliar with the Siebel application) to easily monitor a Siebel OLTP instance and identify performance problems more promptly.

Jonathan Lewis poses a very interesting question:

Have you ever wondered how hard the optimizer has to work to produce a plan ?

Well actually Jonathan, curiously enough I have. Many times, in fact.

Siebel 7.7 (released in 2004) was the first version of Siebel to support the Oracle cost based optimizer (CBO) and pleased a lot of demanding Oracle DBA's who looked contemptuously on the 10 year old rule based optimizer (RBO) technology and had long petitioned Siebel to finally add support for CBO to improve performance, support table partitioning, parallel query, bitmap indexes and lots of other features they wanted to use (i.e learn about).

Inevitably, this major architecture change brought improvements in performance and was generally welcomed by new customers. However, the use of CBO also introduced some new issues for existing, long-standing Siebel customers who upgraded to 7.7 from previous releases.

For example, if you dug very deep with tkprof and ran comparative tests, it was common to observe that parse times were increased with CBO. This was because versions of Siebel prior to 7.7 used the rule based optimizer which follows a fixed set of, 15, err, rules to determine the query plan to be used. Consequently, parsing in RBO is deterministic and relatively quick.

Academics proved that it was actually possible for a well-trained monkey to learn and recite the 15 rules and accurately predict the query plan that Oracle will select. Clever Siebel 2000 developers discovered they could influence the chosen query plan by reciting magic incantations, altering search specifications, sacrificing lambs, nullifying the use of unhelpful indexes and even modifying the ordering of WHERE clauses in the SQL statements.

However. as Siebel finally moved into the 1990's and embraced CBO, things suddenly became very different. The strategy for CBO is to simply give the optimizer as much data as possible so the optimizer can make the 'best' decision regardless. Talented Siebel developers everywhere quaked in their boots and now sacrificed their lambs, praying for a contract renewal.

So whenever Siebel customers complained to me that parse times were too high, I was often tempted to ask them:-

'How many pages of A4 did it take to print out this problematic query with the high parse time ?'

'Err, 16.' 'How many tables are joined ?'

'Well, err, 38.' 'And how many outer joins are in this query ?'

'Well, err, 12.' 'How many indexes are defined on those 38 tables ?'

'Well, err, 189 ?'

'So, if I gave you detailed statistics on those 38 tables together with even more statistics on those 189 indexes together with frequency histograms of all of the columns involved, how long would it take you to parse that query and tell me the best query plan to return the data as quickly as possible ?'

'Well, err, I'm not sure'

'And how long do you think it would it take you to write a computer program to crunch all that data and return the best query plan ?'

'Well, err, I really don't know'

'So do you still think that 3.4 seconds is really too long to parse this statement for the first time and sub-second thereafter ?'.

'Well, err no, I guess not'.

Now I am lucky enough to work for Oracle, I have visibility to a wide variety of complicated CBO related bugs including detailed analysis from Technical Support and Engineering of 10053 trace files.

Consequently, I have nothing but complete respect for the software engineers who develop, maintain, fix and seek to improve the Oracle cost based optimizer.

[ I can thoroughly recommend Jonathan Lewis' book as an excellent introduction with plenty of examples to illustrate how the CBO works. ]

I had the misfortune to visit a very angry customer today. He was having problems loading customer data into Siebel. I sat him down with a nice cup of tea and invited him to tell me all about his woes.

'Well Norman - it's like this you see. We used to be able to load 400,000 contacts per hour. Now performance is absolutely abysmal and it takes 12 minutes to load just 1,000.'

'Hold on. What exactly have you changed since you last loaded 400,000 per hour.'

'Nothing. Nothing at all.'

'Hang on. When were you last able to load 400,000 per hour - exactly ?'

'Err well. It was back in February, you see.'

'And you expect me to believe that you have changed nothing since then ?'

'Yes.'

'OK. I want hard copies of all change controls raised against this system since February and access to Visual Source Safe. I also want the DBA and Siebel administrator here. Now.'

'We haven't got any change controls. We don't use configuration management and I am the DBA, Siebel Admin, systems admin, Web server guru and I also make the tea. We are a small company, you see.'

'I see. That's fine.' (sigh and roll eyes upwards). This could be a long day.

And so the detailed analysis commenced. The control file (IFB) used to load the data. The key Oracle parameters. The hardware of the various servers. All identical to the configuration used back in February.

And he did load 400,000 records per hour. He had the logs to prove it. And now he could type the data in quicker. I know because I saw it with my own eyes. A real mystery.

Siebel's data loading utility is called EIM and uses CBO so time to check the statistics. This client uses Oracle 10g so statistics are automatically gathered by default (essentially gather stale) so all tables have current statistics.

Hold on a minute. All tables apart from EIM_CONTACT and a load of others.

'Why have you dropped statistics on EIM_CONTACT ?'

'We haven't.'

'Oh yes you have.'

'Oh yeah. I remember now. I was playing with a custom script to drop stats on all empty tables. Siebel told me to do this because it affects performance (Alert 1162).'

'Oh I see. How very interesting.'

It transpires that because the interface table is (correctly) truncated post-load, the statistics subsequently got deleted by this script. Kudos to this customer though for humorously naming the script 'dropem.sql'.

Quickly compute statistics on EIM_CONTACT and indexes and performance is miraculously restored.

Well almost. The client forgot to tell me that he had dropped unused indexes on the base tables to accelerate the data load still further. The 'ddlsync' utility that synchronises the Siebel Repository with the physical Oracle schema had subsequently been run which reinstated all the indexes slowing performance down slightly. The indexes are dropped and the original throughput is restored.

Well almost. The client forgot to tell me he had enabled mobile web clients as an experiment for Olof Ericcsson (VP Sales - Scandinavia). So we disable remote transaction logging to eliminate that minor overhead. Throughput is now restored.

So I said my goodbyes and left for the long drive home.

Only, on this occasion, the journey home was very short.

On this occasion, the client happened to be sitting at my kitchen table.

On this occasion, the rack of expensive application/Web/database servers were actually all co-hosted on my very own laptop.

On this occasion, the client who made all the mistakes was me.

There is nothing worse than a blog entry titled Dear Diary - hence the title - but this is my humble submission for One Day in History so please forgive me. Of course, I could author this piece directly on the site but that would mean checking I am within the 650 word limit which is far too much work (unless I use Microsoft Word).

Get up at 7am and unload dishwasher. My one and only household chore but one I perform very well. Breakfast of cereal, tea and orange juice.

Quick scan of work email to see if anything needs my attention. Today I am going to be onsite with a customer and unlikely to have access to email.

Quick scan of Netvibes. The only thing to interest me is the recent update to TailRank. I have occasionally played with various meme trackers (TechMeme, Digg, Reddit, populicio.us) but usually my feeds contain the same stories of interest. I also like popurls as a quick dashboard of current and breaking news.

Today I am visiting a Siebel customer in Ascot. This location is convenient for me (no hotels, no flights, no lengthy drive) and is a shorter journey than to Oracle's offices in Thames Valley Park.

That's strange. For some reason, the traffic in Kingston upon Thames isn't completely gridlocked and I actually cross the bridge over the River Thames in less than 20 minutes. Then I remember why. This week is half-term for some local schools (my son is on holiday) but, much to his glee and her chagrin, my daughter is still at school.

Drive through Sunningdale and marvel at the size of some of the houses (mansions would be a better description) here. Although I keep hearing that Sandbanks in Dorset is the richest area in the UK, Sunningdale must run it a close second.

Arrive at client offices. I am slightly early which is better than slightly late so I make a quick call to an account manager with a ('political not technical') query from a customer about a review from last week.

This customer has a serious data issue in production. On July 22 2006, a 'consultant' advised the customer to configure Siebel to use Universal Time Co-ordinated (UTC). This proposal was a good idea (TM) but poorly executed.

Key elements of the UTC migration were omitted (migration of pre-existing non-UTC data, specification of timezones for each group of users who are located around the world, modification of the database timezone). Apart from that, the conversion to UTC went very well.

The net result was obtuse behaviour and incorrect appointment times observed by different users. The issue is compounded by the fact that this customer also has mobile Web clients who use Siebel on a disconnected laptop and then synchronise with the server. In addition, the Siebel administrator reports that server components (EIM, Workflow) function but now report incorrect submit/start times.

Fortunately, Siebel has a conversion utility (documented in the Global Deployment Guide) to address the issue so I spent some time configuring and testing to fix the data quality issue for the various scenarios.

Get home. Eat tea. Settle down to watch United versus Copenhagen. Discover that the wife is going out (how very inconsiderate) and I have pop out to collect my daughter from trampolining.

Count words using Google Docs - 560.

Someone recently asked me at a dinner party: So, Norman, tell me what you do in life ?. I spontaneously replied: I help people read books. The lady (for it was a she) exclaimed: Oh how absolutely fabulous. You are a teacher. Err, well, no. I actually work in IT. Oh I see. You work in training. Why didn't you just say so ?' 'Err, well no. I am a sort of IT consultant'.

Anyway, after an embarrassing stony silence, thankfully I managed to steer the conversation to the safer domain of the wide range of choices for secondary school education in our locality. This fascinating subject occupied us right through until the desert and coffee were served.

But the point I was trying to make was that Siebel and Oracle are incredibly large, complicated, wide ranging software products. I have worked with Siebel for three years and Oracle for a little longer but there are still so many areas and modules in both products that I have no practical experience of whatsoever.

I remember once reading Tom Kyte stating that he did not have access to the Oracle source code nor did he did not have a hotline to RDBMS engineering. The basis of his wealth of extensive Oracle knowledge was primarily the documentation set. I remember being hugely impressed by this simple statement. [ Sorry I did look but failed to locate the reference ]

I am a Siebel 'consultant' trying to help people use Siebel more effectively. Most of the information needed to help customers use Siebel more effectively is actually contained in the documentation. The only problem is that the 'documentation' is simply overwhelming as it includes the manuals, FAQ's, Alerts, Release Notes, Service Requests etc etc.

I have a couple of advantages: Firstly, I am continually exposed to a wide variety of different Siebel related issues day after day so I so have a degree of experience of real-world problems (and hopefully the resolution).

Secondly, and more importantly, I do have access to a network of highly talented, intelligent individuals with far more experience and intelligence than yours truly. Now this wouldn't be an advantage unless that group of people were prepared to share their knowledge and I am pleased to say that they are. This isn't necessarily true at all companies I have worked for.

Normally, I lug my heavy laptop, hanging over my shoulder, attached to my body like a young helpless infant, all around Europe. Today I was in Stockholm and the weather was unusually hot (30'C). To reach the office, I had to take a train and a tube in the morning rush hour. Consequently, I left the laptop behind in the hotel and arrived onsite free from back pain and feeling blissfully liberated.

I told the customer that we would purely be using the public documentation that is freely available to me and him. No hidden cheat-sheets, no private internal emails, no top tips from engineering. He was impressed (I think).

Then, of course, inevitably, we hit a very obtuse, bizarre problem, neither of us had encountered before so it was time to make another call on that network.