The million dollar Oracle surprise – Part 1

I decided to write this article to share some of the surprises we have uncovered while carrying out Oracle Licensing projects with our customers. A million dollars may seem like a lot and you may think that this is the size of problems in larger organisations. However, the figure of a Million Dollars is based on a single 2 node cluster with 56 cores per node, a typical specification these days.

The calculation is as follows:

2 x 56 Cores = 112

Multiplied by a Core factor of 0.5 = 56 licensable Oracle Processors.

Take a typical database option such as Spatial which has a list price of  $17,500

$17,500×56=$980,000  which doesn’t leave much change from a million dollars.

If we were to take the database itself which has a list price of $47,500 we are looking at $2.66 million.

This doesn’t take into account back support which you would also be asked to pay if this were found in an audit. The bottom line is that nobody is too small or large to ignore their Oracle licensing. This article will go through six areas where we typically find the surprises. Within each area I will explain how these surprises arise giving real life examples (don’t worry we won’t name names) of how they were created and finally we will recommend steps that you can take to manage these risks within your organisation.

The six areas are as follows:

1. Unintentional Option or Feature Usage
2. False Positives
3. Beware of Patches
4. Virtualisation Surprises
5. Over Specified or Shared Servers
6. The limits of the Unlimited License Agreement

I plan to write this in two parts, finishing later this month as I need to fit this in with the day job. This post will cover the first three areas and  part 2 will cover the last three points along with some conclusions and recommendations.

 1. Unintentional Software or Feature Usage

Unintentional Software Activation: Oracle don’t restrict the installation or usage of their software and so it is easy for organisations to find that they are using software that they are not licensed for. Nearly all Oracle Software is available for download from the Oracle website and can be installed for development, test and evaluation purposes for free. On top of this multiple products can be installed from a single download, for example the database can install standard or enterprise and will generally install most options when you pick enterprise.

This downloaded software is governed by the OTN Agreement. http://www.oracle.com/technetwork/licenses/standard-license-152015.html

There is no difference between the software downloaded from Technet and what is shipped by Oracle Support to customers. Often we see copies of various products on VMWare virtual machines or Desktops in organisations for which they have no license. The reasons for this can be a lack of clarity on exactly what an organisation is licensed for and the current usage and limits of that license. We’ll talk a little more about the misleadingly named unlimited license agreement in the next part of this blog, but no organisation we have worked with has managed to control the installation of Oracle software completely.

Unintentional Feature activation: The above example of software installation creating compliance issues will be no surprise as someone has intentionally installed it. However,  what if you install a product, but by using specific features of the installed product you create a compliance risk for your organisation. Surely this can’t happen? It can and does all the time. The main culprits are database options, but also within the middleware products there are multiple features that can catch you out. Let’s look at some specific examples of how this can happen.

There are various ways that DBA’s can cause usage of Advanced Compression, and this happens in complete silence – no flashing lights, warnings or anything – just a record stored in your database to let Oracle know in the event of an Audit that you used it and are liable for all the cores in the cluster that this database is in (plus back support!)

One of the most common triggers for this is Data Pump which is Oracle’s high performance version of the old export import utilities. If you use Data Pump with compression you have used the Advanced Compression feature and are liable for a license on that database.  The same goes for Advanced Security when Data Pump is used with encryption. More on Advanced Security later. We have seen examples where a single usage of Data Pump from over 5 years ago has created a significant licensing compliance risk.

Another example we see very often is the database tuning pack which again is an additional, licensable option. This can easily be triggered by enabling AWR or ADDM database features.   These are an evolution of the Oracle Stats Pack functionality which is still available for free, but if you use these features you need a license for the  Diagnostics and Tuning packs which are $7500 and $5000 respectively per core.  That is almost $400K  on a typical (32 core) server to understand why your database is running slowly!

Weblogic: Taking a Weblogic example, you have one installer for all Editions of Weblogic as well as some of the optional extras.  This means you need to be careful that you only install the components that you are licensed to install and that no additional features are enabled or used post installation. It is all too easy to enable clustering or other enterprise features on Standard edition, thereby rendering yourself non-compliant.

Middleware analysis is very complex and Oracle have complex scripts that will search your servers to look at the configuration of your Weblogic instances to see exactly which features you are using.  This analysis is one of the most common tasks we are asked to do for our clients as it is very labour intensive to analyse without some automation.

These are just a few of examples, but there are many more – just look at the Oracle price list to see the potential.  We recommend that all options are uninstalled from databases by default to ensure that they cannot be used unintentionally.  This is not always possible or practical, but it is important to educate your development and administration staff on these risks and take steps to minimise them.

This is a complex area and there is no way to completely stop unintentional usage as the features and rules are always changing and differ from product to product and version to version. This said we recommend that organisations acknowledge that this will happen and have policies and processes in place to minimise the risks which can be very large if you get an oracle license audit or review. These features are time bombs waiting to be discovered. The sooner you find them the more time and options you have to resolve them with minimal financial impact.

 

2. False Positives

There are many tools out there to discover various Oracle products and options and as there are no clear or documented  ways of determining if a product is installed and in use.  Due to this there are often false positives.  There is no easy solution to this problem without getting into the minute detail of some of the products and their signatures. Even when you do there are situations where you can’t tell the difference between an included feature and an additional cost option. 

Examples of these include:

• Spatial being flagged when only locator is being used. In older database versions Oracle have no way to tell between the two in terms of usage and Spatial and Locator are installed by default with Enterprise edition.
• Advanced Security is flagged in DBA Feature Usage incorrectly when op$ accounts are used in 10.2
• Multi-Tenanted is flagged when only one PDB exists rather than 2.
• Advanced Compression or Security when Data Pump is used.

We always recommend that full analysis is done and that organisations ensure that the evidence being used is accurate. secondary checks are also recommended to ensure that these false positives are identified. For many of these there are preventative measures that can be taken to ensure certain options are either uninstalled or disabled to ensure that they are not used.

 3. Beware of patches and upgrades

We have already talked about unintentional usage of features but what if your policies and procedures are correct but the application of a patch or an upgrade causes a feature to be used or enabled? Think this can’t happen? Do a search on Oracle support. We have seen examples of patches upgrading  an Oracle Standard edition to Oracle Enterprise Edition. It took a lot of head scratching and ultimately investigation of logs to establish what happened in this case and for a long time it was assumed that it was installed incorrectly from the start, which was 2004 – think about all that back support if it had been so.

The symptoms vary and although we have seen a lot we see new ones all the time. The following are some of the examples we have seen.

• Patches that upgrade a database from standard to enterprise. Yes, search support if you don’t believe us.
• Patches and updates that use an option as part of the update process, these are things like advanced compression or security, but some can leave demo accounts that install OLAP or other features installed.
• Patches that re-install or re-enable options that have been previously removed or disabled.

Administrators are continuously maintaining the software and instances under the direction of Oracle support. The reality is bugs exists in software and mistakes are made but don’t assume that Oracle doesn’t make some too.

Unfortunately to establish and ultimately prove that a patch or upgrade is responsible; is very hard. However, Oracle provide the solution too in their LMS Inventory scripts. We recommend that organisations regularly run these and archive them to log the position at various points in time. This will allow you to pinpoint when a change occurred that impacts your license position. Not only will it do this but it will allow you to provide this evidence to Oracle in the format they support.

We recommend that before and after any patches or upgrades, you run these scripts to ensure you have evidence that can be used in the event that an unexpected change occurs. It’s also worth understanding the outputs from these scripts to determine what options and packs are in use on your databases. You can obtain these scripts from Oracle support and we recommend that you keep them updated as they change every year.

Thanks for reading Part 1 of this blog. In Part 2, I will be covering the last three topics I mentioned plus some general conclusions and recommendations.

If you would like to discuss this or any aspect of your Oracle licensing then please feel free to contact us;