Posts in category "oracle"

I own up. Its a fair cop. It is 27 years and 5 months since my last confession. Since then, I have sinned, Holy Father. In fact, I have committed an absolutely heinous crime. Please forgive me for I have submitted multiple random guesses.

May the high priests (and priestesses) of the BAAG movement, please have mercy on my wretched soul.

Recently, I have been guilty of supporting and even proposing a multitude of 'any guesses', all of which were proposed as possible solutions to solve a critical system problem on a production system:

1. I failed to cough and splutter in an effort to stifle my laughter when a manager suggested that 500 European users adopted shift working to clear the backlog.
2. I didn't shout down a ludicrous proposal to reboot the Siebel Enterprise every 4 hours in order to maintain some level of service.
3. I wrote on a whiteboard that consideration should be given to reinstating the previous version of the application even though this in itself was risky, time-consuming and unlikely to address the root cause.
4. I even proposed patching to Oracle 10.2.0.3 with no supporting evidence whatsoever.
5. I shuffled nervously and blushed when the customer asked 'Who do we escalate to when you fail to fix this problem ?'
6. I watched in silence as the SAN man was stood against a wall and pelted with questions as senior management all pointed fingers in his direction.
7. I failed to stand up for righteousness and technical purity as runaway, rogue sessions were maliciously and arbitrarily terminated by an Oracle DBA.
8. I failed to suggest running a trivial SQL to determine index fragmentation and stood by as 74 indexes were needlessly reorganised.
9. I watched helplessly as a systems administrator claimed the problem was 'definitely in the underlying disk I/O subsystem' simply because 8,000 operations a second were being performed.
10. I failed to raise my eyes skywards and embark on a spontaneous two hour training session with a DBA who claimed the 'buffer cache hit ratio was fine' and the problems only started once users were allowed onto the system.

My only defence is that all of the above 'any guesses' created enough of a diversionary smokescreen to buy me enough time, alone in a darkened room, to analyze multiple Statspack reports, reproduce the majority of the problems in SQL*Plus and then prove the behaviour was improved when statistics on empty tables were dropped and two additional indexes created.

This lucky 'guess' miraculously restored performance and stability. So you see, 'any guess' is not always such a bad strategy, after all.

This Bluehost powered blog is now FastCGI enabled. You may find the site faster but, then again, hyperlinks may return completely random articles. You will probably struggle to tell the difference.

Which reminds me of a funny story...

I am currently working in the North East on an 'escalation'. An 'escalation' is characterised by frequent, lengthy conference calls with bi-hourly status updates, the phrases 'high-profile', 'alternative technical solutions', 'severe degradation', 'impacting the bottom line', 'CIO visibility', 'options for the weekend' and the perennial favourite: 'any progress yet ?'

Thus far I have managed to resist the temptation to reply: 'No. Sorry. No progress yet because I have been in meetings and conference calls since I arrived onsite and I haven't even been offered a coffee yet, let alone actually touched a keyboard.'

Anyway, I digress. All of these factors combine for a relatively stressful working environment.

This lunchtime, during a scheduled outage of 15 minutes to stop and restart Oracle to change a static 9i parameter (optimizer_index_cost_adj from 1 to 10 for the technically minded), three highly skilled, overpaid technical people and a project manager were overseeing this vital, complicated and potentially life saving modification.

The application servers were shutdown so those pesky users could no longer use the system. We looked wistfully at a AIX 'topas' screen showing 'CPU: 0%, User: 0%, System: 0%, Disk 0%, Wait: 0%' and took a screenshot for posterity.

We paused to savour the moment. There was a brief moment of quiet contemplation and tranquility. Then the DBA piped up: 'Shall we wait for 1 minute ?'

The DBA then fired up Enterprise Manager. We all doubled checked the password credentials and the target database. We held an impromptu video conference call to get approval to press the 'Login' button.

We doubled checked that password for SYS was secure, non-intuitive and, err, different for the DEV, TEST, UAT, STAGE, QA, TRAIN and PRODUCTION environments. Of course, it was. What sort of organisation responsible for Oracle outsourcing would do anything different ?

The moment had finally arrived. Hours of detailed analysis. Hours of pouring over Statspack Level 7 reports and query plans. Hours of talking to the mysterious 'SAN Man'. Hours of scouring SupportWeb and Metalink.

Hours of investigating helpful (but ultimately unhelpful) suggestions from people who should not have been involved. Hours of going deep down into dark rat holes. Hours of early starts, late finishes, no coffee and plenty of red herrings.

The time had finally come to change the parameter. We stand, poised, on the verge of greatness.

I broke the tense silence: 'OK. The Oracle parameter we need to change governs the behaviour of the Oracle optimizer. We are seeing an expensive 'INDEX FULL SCAN' on the customer table which generates million of logical I/O's, physical disk reads and brings the database server to its knees. We are trying to encourage Oracle to use 'INDEX RANGE SCAN' on the driving, intersection table instead which is 300 times more efficient.'

'We are unable to add hints. We are unable to modify the SQL generated by the black box so we are going to change a key Oracle setting for CBO. I have the Change Control in my hands. We are actually going to change an Oracle parameter.'

A hush descends. The DBA observes another minute silence and expectantly clicks 'Instance - Configuration'. With a little prompting, he clicks 'SPFile' instead of 'Running'.

I continue: 'The parameter we are about to change is called 'Go' 'Underscore' 'Faster'.

The DBA then scrolls down to the 'G' section.

There is a period of a superlative 3 second silence followed by 'Sorry - what did you say: 'Go' 'Underscore' 'Fas...' followed by three people (poor project manager) laughing uncontrollably.

The door opens: 'Any progress yet ?'

You never press ‘Publish any more
When I read your feed
Theres no articles like before
In your blog
You're trying hard not to show it
But baby, baby I know it
You've lost that blogging feeling
Oh, that blogging feeling
You've lost that blogging feeling
Now it's gone, gone, gone
Oh Tom

Now there's no comments
In your blog when I read you
And Tom you're forgetting to comment
Little things I do
It makes me just feel like crying baby
‘Cause Tom, something beautiful's dying

You've lost that blogging feeling
Oh that blogging feeling
Bring back that blogging feeling
Now it's gone gone gone
And I can't go on
No-oh-oh

Baby baby I get down on my knees for you
If you would only link to me like you used to do
We had a blog
A blog, a blog you don't find every day
So don't…don't…don't let it slip away

You've lost that blogging feeling
Oh that blogging feeling
Bring back that blogging feeling
Now it's gone gone gone
And I can't go on
No-oh-oh

With apologies to The Righteous Brothers and Tom Kyte.

Another baby when my wife is 57 years old ? - discuss.

My job title when I joined Siebel in April 2003 was 'Junior Architecture Specialist'. Four years later, it transpires that I am now officially entitled to call myself 'Principal Architecture Specialist'.

However, I have elected not to change my business cards and .signature as I am simply not worthy. Please remember that I have worked with some very talented individuals (past and present).

In any case, who needs more boxes of business cards gathering dust ? I rarely dispense my business card to any client as I much prefer email as the sole communication channel. This also avoids the possibility of a customer calling me on Sunday evening to assist with a Service Request escalation or go-live support. After all, 'Antiques Roadshow' is absolutely sacrosanct in our household.

Secondly, such a deluxe, overblown, Rolls-Royce description may set the client expectations unreasonably high and this may lead to dashed hopes and bitter disappointment (especially when dealing with me).

So when I cross the threshold of the modern glass-fronted offices, extracting my contact details and preparing my firm handshake, I normally introduce myself with 'Hi, I'm Norman Brightside from Expert Services. I know a little about lots.'

To be fair, this statement is unerringly accurate and leaves me with the perennial 'Get Out of Jail Free' card: 'Oh I'm sorry - but that's an area of Siebel I am not intimate with' (which I tend to use a lot).

Last March, following the Oracle takeover, our group was going to be re-branded as 'Service Delivery Engineers'. All Siebel employees were also eligible for higher rate tax relief on any dental surgery and liposuction, if required.

I was abroad in Sweden and didn't really care - after all, what's in a name ? - Expert Services actually do 'deliver services' to customers although we are not (software) engineers in the traditional sense.

However, some of my colleagues fiercely resisted this change with a vengeance (think 1990 Poll Tax riots) which left me rather bemused. The standing joke was that we would tell customers; 'Sorry I haven't got that part on the van' and suck through our teeth 'Hmm - this isn't gonna be cheap' when faced with a thorny performance problem.

I honestly wouldn't care if Oracle called me 'Overpaid, ugly code monkey' as long as my salary and benefits were preserved. In fact, when I was a mercenary contractor, I toyed with using this job title as my official occupation on my passport.

Surprisingly, the revolutionaries prevailed and the corporation ruled that we could preserve our job titles or upgrade, at no extra cost, to 'Electrolux repair man', if we so desired.

Twelve months on, my contrarian instincts are twitching so what do you think ?

• Service Delivery Engineer
• Architecture Specialist
• Technical 'Consultant'
• Amusing alternative

Dust off that blogroll. Highlight your best technical posts. Update that photo in the About tab. Polish the colours on your theme. Prime the hit counter and prepare for an invasion from the Web 2.0 A listers.

Justin Kestelyn (Oracle Technical Network) sparked some an interesting and though provoking discussion when he puzzled over the relatively low profile of Oracle Corporation in the Web 2.0 community.

Robert Scoble picked up the thread (twice) and there are some interesting comments. Certainly, I'd love to see a Scoble podcast from Oracle Corporation.

I don't know enough about what Oracle's rivals (IBM, Sun, HP, SAP, ~~Siebel~~) are doing out there in the blogging community or how these companies are perceived by the Web 2.0 community to know whether Oracle is hard done by.

That said, a couple of points on Justin's original post

'Oracle's aggressive support of blogging'

As an Oracle employee, while Oracle encourages and supports the blogging efforts of both employees and non-employees I think 'aggressive' is probably overstating the case. For example, Oracle don't offer a hosted blogging platform for aspiring authors merely a listing in a (albeit high profile) directory.

' rather large blogging community'

For a company of 70,000 employees worldwide, does a blogroll containing 61 employee blogs, 10 executive blogs and 100 blogs from technical users truly constitute a 'large blogging community' ?

It would be interesting to know how the number (in percentage terms) of Microsoft and Google employees actively blogging compares with Oracle.

I am an Oracle employee and I have a neglected (internal eyes only) corporate blog only and this blog for my own personal outpourings. I have also occasionally toyed with the idea of applying for a corporate blog. However, to date, I have always resisted this temptation because although I am positive that a listing on blogs.oracle.com would drive a lot of traffic and boost my ego, maintaining a corporate blog carries a great deal of responsibility.

I am not referring to the Oracle guidelines governing content on a corporate sanctioned blog as these are common sense and perfectly reasonable. In fact, I am already obliged to abide by those guidelines here on this personal blog.

Authoring an 'official' Oracle blog would immediately create a wealth of (self-imposed) pressure on me to maintain that blog and keep adding technical, well researched, accurate, interesting and valuable content.

In my current role, I spend most of my working time working for customers. I simply do not have the spare time to spend maintaining a corporate blog.

'Or maybe I shouldn't even care !'

Justin's closing paragraph is interesting. I am an Oracle employee and I also hold Oracle stock. Consequently, I am more concerned with Oracle product development and the stock price than how many engineers are blogging this week and whether Oracle have been invited to participate or host the latest 'Lunch 2.0'.

The climax of the football season is currently dominating my life somewhat. However, I feel obliged to inform the world of my exciting plans for the next fortnight.

Following next week's UK Bank Holiday, the scheduling goddess informs me me that I will be doing a 'brown bag'. This means that next Tuesday will see me wandering aimlessly around the shopping precinct in Reading, spontaneously shouting at shoppers while clutching four cans of Special Brew in a brown bag.

For the remainder of the week, I will be returning to the scene of last years summer holiday - Norway. Again, the work calendar informs me I will be spending 8 days embroiled in what is tersely described as a 'deep dive'.

While I appreciate the break from work and the respite from staring blankly at a computer screen, I do have some reservations about this 'deep dive' activity. Bear in mind that I very nearly drowned whilst hiring a motor boat on the Norwegian fjords last year.

Still, it will be a fantastic opportunity to learn all about the fascinating world of scuba diving and hopefully realise a lifelong ambition and gain my Padi Open Water qualification. This means I will be able to stop boring people with my detailed analysis of the football season and my offspring's sporting prowess and academic abilities.

Now, armed and dangerous with a Padi, I will fully empowered to participate in prolonged and tedious dinner party conversations about the joys of scuba diving and the pros and cons of the crystal clear, warm water in Egypt versus a purpose-built, 20 metre, freezing swimming pool in Plymouth naval base.

Many years ago, I managed a set of Oracle databases for various clients.

However, I was not an Oracle DBA. I was an Unix/C developer who happened to progress to Pro*C, PL/SQL and some ETL projects for data warehouses.

I was an mediocre development DBA because I was a mediocre developer and I had a keen interest in performance tuning i.e. I was (am remain) a 'glory hunter'.

I was not a production DBA because I didn't have the training, experience and discipline required for change controls and saying 'No'.

One day, the CEO of our 18 man Internet startup asked me to deliver a short presentation to the rest of the company what my team (me and my junior) did all day.

I put up a slide that said:

• Data Availability

I think a clever new media colleague helped me to add 'transitions' to add two more bullets.

• Data Availability
• Data Availability

Ironically, this wasn't what we did which was covered on the next 34 slides. We did absolutely everything: fixing code, reloading data, backups, analysis, ETL, extracts, PL/SQL, performance tuning, long hours, recovery, testing, code reviews, cloning, documentation, nervous breakdowns, monitoring, upgrades, configuration management, severed relationships, recruitment, management and coffee.

However, data availability is precisely the only thing we should have been doing.

Data availability is the sole responsibility of an Oracle production DBA.

It is to ensure that data is available at all times to all users.

It is not to tune queries from 37 seconds to sub-second.

It is not to experiment with parallel query and parallel DML to tune a third party application.

It is not to ensure that you can clone 'ALL_OBJECTS' in less than 17 seconds.

It is not to ensure you can load 184 million records in 72 hours.

It is to ensure that you can restore data from a catastrophic failure.

It is to ensure you can restore service following an expected failure.

It is to ensure you can restore from the backups deposited in secure off-site storage.

It is to ensure that the file system holding the archive logs never fills up.

It is to ensure that you can have a coherent plan to restore service after any unexpected event.

It is to ensure that you take responsibility and say 'Yes. That is down to me'.

Holy Father. It is almost 31 years since Citys last trophy. Since then, I have:

• Failed to install Oracle 10gR3 Grid Control to experiment with the Siebel Management Pack.
• Failed to configure bridged networking on my Siebel 8.0/Oracle 10g/RedHat VMware environment. Consequently, I have just deleted the complete 20GB environment in a fit of pique.
• Nearly exploded at the woeful response to a ticket with Bluehost asking why two functioning cron jobs were a) removed and b) failed to work when reinstated.
• Failed to resolve a trivial but incredibly annoying problem with AVG mail filter.
• To top it all, I don't even know the difference between MB and Mbps.

'My dear son. You are indeed a worthless sinner. May the Lord may mercy on your soul. To seek forgiveness and redemption, please say three Hail Marys, one Our Father and request a transfer to pre-sales.'

I am currently trying to assist with a couple of long standing Service Requests. This type of work is interesting because the issue is normally pretty deep rooted and complicated. These escalations are also very challenging because lots of very intelligent people have already spent a lot of man-hours investigating the problem.

Anyway, one of the first steps is to review associated Service Requests and try to determine whether they are indeed related to the issue under investigation.

The opening paragraph of one such SR contained this bold assertion from the customer: 'As you can see, we have populated the interface tables correctly and EIM still doesn't work.'

This was a interesting statement and was directly related to the problem I was looking at. This statement went unchallenged by Technical Support so I took it at face value and continued my analysis.

There was further empirical evidence that this data load would fail as changes to the corresponding meta-data in the Siebel Repository had been made in version 7.8. A user key definition on this table had been inactivated which was previously present in 7.5. Interesting.

A few hours later, I returned to this SR as I was about to actually try loading data into the same Siebel tables. This seemed rather pointless if it wasn't going to work. However, I started from scratch, read the documentation and created my own simple test case for a single record.

Sure enough, the customer was correct. Even though, I had populated the correct columns in the correct interface table with the correct data, Siebel failed to populate the target tables.

However, when I reviewed the EIM log files more carefully, the errors were the conventional (foreign key lookup failed) type. I fixed the test data, re-ran the load script and sure enough data appeared in the target tables and was visible in the Siebel application.

So, the customer was actually mistaken and I suspect his bold assertion including the four little words 'As you can see' also misled the technical support engineer.

Last night, my son told me that playing World of Warcraft was much, much quicker since the recent broadband upgrade from 4Mb to 10Mb. This made sense as online gaming presumably needs lots of bandwidth and Virgin customer services told me the upgrade would be effective from last Friday.

Out of curiosity, I downloaded a 100MB file (twice) and looked at the download speeds which were in the range 360-390 KB/sec. Quick but less than you would expect for a 10Mb connection.

I called my friends at Virgin Media to check the status of my broadband upgrade. Sure enough, contrary to what I was told, the 10Mb upgrade will only be activated on 23 March when the engineer installs the V+ box.

So, unless you see it with your own eyes, it didn't happen.

Oracle blogs I read by Technorati ranking. Cue Led Zeppelins Whole Lotta Love.

1. Life After Coffee - Jon Emmons - Rank: 18,499 (405 links from 190 blogs)
2. The Tom Kyte Blog - Tom Kyte - Rank: 24,521 (345 links from 145 blogs)
3. Oracle BI and DW - Mark Rittman - Rank: 31,939 (458 links from 112 blogs)
4. Oracle Scratchpad - Jonathan Lewis - Rank: 44,328 (242 links from 83 blogs)
5. Eddie Awad's Blog - Eddie Awad - Rank: 60,122 (175 links from 62 blogs)
6. Doug's Oracle Blog - Doug Burns - Rank: 83,319 (216 links from 46 blogs)
7. Kevin Closson's Oracle Blog - Kevin Closson - Rank: 87,506 (100 links from 44 blogs)
8. The Oracle Sponge - David Aldridge - Rank: 102,376 (72 links from 38 blogs)
9. The Oracle-Base Blog - Tim Hall - Rank: 112,139 (57 links from 35 blogs) 10 So What Co-Operative - Jeff Hunter - Rank: 154,372 (62 links from 26 blogs)
10. Pete-s Random Notes - Peter Scott - Rank: 161,008 (92 links from 25 blogs)
11. Radio Free Tooting - Andrew Clarke - Rank: 168,201 (70 links from 24 blogs)
12. Oracle 10g Blog - Chris Foot - Rank: 168,201 (51 links from 24 blogs)
13. Oracle WTF - William Robertson - Rank: 194,372 (70 links from 21 blogs)
14. Dizwell Informatics - Howard Rogers - Rank: 194,372 (61 links from 21 blogs)
15. OracleBlog - Robert Vollman - Rank: 194,372 (44 links from 21 blogs)
16. Oracle Stuff I should have known - Andy Campbell - Rank TBD (55 links from X blogs)
17. We Do Not Use Blogs - Mogens Norgaard - Rank: 259,194 (26 links from 16 blogs)
18. Laurent Schneider OCM - Laurent Schneider - Rank: 322,022 (176 links from 13 blogs)
19. Oracle Newbies Blog - Lisa Dobson - Rank: 322,022 (29 links from 13 blogs)
20. OraMoss Oracle - Jeff Moss - Rank: 347,530 (150 links from 12 blogs)
21. Confessions of a database geek - Beth - Rank: 347,530 (32 links from 12 blogs)
22. Oracle Alchemist - Steve Karam - Rank: 413,015 (13 links from 10 blogs)
23. The Eric S. Emrick Blog - Eric Emrick - Rank: 413,015 (12 links from 10 blogs)
24. Oracleoid Blog - Alex Gorbachev - Rank: 656,741 (27 links from 6 blogs)
25. BobaBlog - Robert Baillie - Rank: 656,741 (7 links from 6 blogs)
26. Igor's Oracle Lab - Gary Myers - Rank: 766,996 (6 links from 5 blogs)
27. DBA's R US - Nuno Souto - Rank: 1,220,048 (11 links from 3 blogs)
28. Optimal DBA - Daniel Fink - Rank: 1,220,048 (11 links from 3 blogs)
29. Oracle on Windows - Edward Whalen - Rank: 1,724,033 (3 links from 2 blogs)