Kingsley Uyi Idehen's Weblog

In recent times a lot of the commentary and focus re. Virtuoso has centered on the RDF Quad Store and Linked Data. What sometimes gets overlooked is the sophisticated Virtual Database Engine that provides the foundation for all of Virtuoso's data integration capabilities.

In this post I provide a brief re-introduction to this essential aspect of Virtuoso.

What is it?

This component of Virtuoso is known as the Virtual Database Engine (VDBMS). It provides transparent high-performance and secure access to disparate data sources that are external to Virtuoso. It enables federated access and integration of data hosted by any ODBC- or JDBC-accessible RDBMS, RDF Store, XML database, or Document (Free Text)-oriented Content Management System. In addition, it facilitates integration with Web Services (SOAP-based SOA RPCs or REST-fully accessible Web Resources).

Why is it important?

In the most basic sense, you shouldn't need to upgrade your existing database engine version simply because your current DBMS and Data Access Driver combo isn't compatible with ODBC-compliant desktop tools such as Microsoft Access, Crystal Reports, BusinessObjects, Impromptu, or other of ODBC, JDBC, ADO.NET, or OLE DB-compliant applications. Simply place Virtuoso in front of your so-called "legacy database," and let it deliver the compliance levels sought by these tools

In addition, it's important to note that today's enterprise, through application evolution, company mergers, or acquisitions, is often faced with disparately-structured data residing in any number of line-of-business-oriented data silos. Compounding the problem is the exponential growth of user-generated data via new social media-oriented collaboration tools and platforms. For companies to cost-effectively harness the opportunities accorded by the increasing intersection between line-of-business applications and social media, virtualization of data silos must be achieved, and this virtualization must be delivered in a manner that doesn't prohibitively compromise performance or completely undermine security at either the enterprise or personal level. Again, this is what you get by simply installing Virtuoso.

How do I use it?

The VDBMS may be used in a variety of ways, depending on the data access and integration task at hand. Examples include:

Relational Database Federation

You can make a single ODBC, JDBC, ADO.NET, OLE DB, or XMLA connection to multiple ODBC- or JDBC-accessible RDBMS data sources, concurrently, with the ability to perform intelligent distributed joins against externally-hosted database tables. For instance, you can join internal human resources data against internal sales and external stock market data, even when the HR team uses Oracle, the Sales team uses Informix, and the Stock Market figures come from Ingres!

Conceptual Level Data Access using the RDF Model

You can construct RDF Model-based Conceptual Views atop Relational Data Sources. This is about generating HTTP-based Entity-Attribute-Value (E-A-V) graphs using data culled "on the fly" from native or external data sources (Relational Tables/Views, XML-based Web Services, or User Defined Types).

You can also derive RDF Model-based Conceptual Views from Web Resource transformations "on the fly" -- the Virtuoso Sponger (RDFizing middleware component) enables you to generate RDF Model Linked Data via a RESTful Web Service or within the process pipeline of the SPARQL query engine (i.e., you simply use the URL of a Web Resource in the FROM clause of a SPARQL query).

It's important to note that Views take the form of HTTP links that serve as both Data Source Names and Data Source Addresses. This enables you to query and explore relationships across entities (i.e., People, Places, and other Real World Things) via HTTP clients (e.g., Web Browsers) or directly via SPARQL Query Language constructs transmitted over HTTP.

Conceptual Level Data Access using ADO.NET Entity Frameworks

As an alternative to RDF, Virtuoso can expose ADO.NET Entity Frameworks-based Conceptual Views over Relational Data Sources. It achieves this by generating Entity Relationship graphs via its native ADO.NET Provider, exposing all externally attached ODBC- and JDBC-accessible data sources. In addition, the ADO.NET Provider supports direct access to Virtuoso's native RDF database engine, eliminating the need for resource intensive Entity Frameworks model transformations.

Related

• Attaching ODBC or JDBC accessible Relational Tables to Virtuoso
• Using an HTML based Wizard to Generate RDF based Linked Views over Relational Tables
• Screencast Demonstrating Wizard based generation of RDF based Linked Data Views Part 1
• Screencast Demonstrating Wizard based generation of RDF based Linked Data Views Part 1
• Generating RDF based Linked Data from non RDF based Web Resources via the Sponger
• Building ADO.NET based Entity Frameworks Views over Relational Data
• Building Silverlight Rich Internat Applicaitons using ADO.NET, Entity Frameworks, and RDF based Linked Data.

Here are 5 powerful benefits you can immediately derive from the combination of Virtuoso and Amazon's AWS services (specifically the EC2 and EBS components):

1. Acquire your own personal or service specific data space in the Cloud. Think DBase, Paradox, FoxPRO, Access of yore, but with the power of Oracle, Informix, Microsoft SQL Server etc.. using a Conceptual, as opposed to solely Logical, model based DBMS (i.e., a Hybrid DBMS Engine for: SQL, RDF, XML, and Full Text)
2. Ability to share and control access to your resources using innovations like FOAF+SSL, OpenID, and OAuth, all from one place
3. Construction of personal or organization based FOAF profiles in a matter of minutes; by simply creating a basic DBMS (or ODS application layer) account; and then using this profile to create strong links (references) to all your Data silos (esp. those from the Web 2.0 realm)
4. Load data sets from the LOD cloud or Sponge existing Web resources (i.e., on the fly data transformation to RDF model based Linked Data) and then use the combination to build powerful lookup services that enrich the value of URLs (think: Web addressable reports holding query results) that you publish
5. Bind all of the above to a domain that you own (e.g. a .Name domain) so that you have an attribution-friendly "authority" component for resource URLs and Entity URIs published from your Personal Linked Data Space on the Web (or private HTTP network).

In a nutshell, the AWS Cloud infrastructure simplifies the process of generating Federated presence on the Internet and/or World Wide Web. Remember, centralized networking models always end up creating data silos, in some context, ultimately! :-)

I am pleased to announce the immediate availability of the Virtuoso ADO.NET 3.5 data provider for Microsoft's .NET platform.

What is it?

A data access driver/provider that provides conceptual entity oriented access to RDBMS data managed by Virtuoso. Naturally, it also uses Virtuoso's in-built virtual / federated database layer to provide access to ODBC and JDBC accessible RDBMS engines such as: Oracle (7.x to latest), SQL Server (4.2 to latest), Sybase, IBM Informix (5.x to latest), IBM DB2, Ingres (6.x to latest), Progress (7.x to OpenEdge), MySQL, PostgreSQL, Firebird, and others using our ODBC or JDBC bridge drivers.

Benefits?

Technical:

It delivers an Entity-Attribute-Value + Classes & Relationships model over disparate data sources that are materialized as .NET Entity Framework Objects, which are then consumable via ADO.NET Data Object Services, LINQ for Entities, and other ADO.NET data consumers.

The provider is fully integrated into Visual Studio 2008 and delivers the same "ease of use" offered by Microsoft's own SQL Server provider, but across Virtuoso, Oracle, Sybase, DB2, Informix, Ingres, Progress (OpenEdge), MySQL, PostgreSQL, Firebird, and others. The same benefits also apply uniformly to Entity Frameworks compatibility.

Bearing in mind that Virtuoso is a multi-model (hybrid) data manager, this also implies that you can use .NET Entity Frameworks against all data managed by Virtuoso. Remember, Virtuoso's SQL channel is a conduit to Virtuoso's core; thus, RDF (courtesy of SPASQL as already implemented re. Jena/Sesame/Redland providers), XML, and other data forms stored in Virtuoso also become accessible via .NET's Entity Frameworks.

Strategic:

You can choose which entity oriented data access model works best for you: RDF Linked Data & SPARQL or .NET Entity Frameworks & Entity SQL. Either way, Virtuoso delivers a commercial grade, high-performance, secure, and scalable solution.

How do I use it?

Simply follow one of guides below:

• Using Visual Studio 2008 & Virtuoso to build an Entity Frameworks based Windows forms application
• Using Visual Studio 2008 & Virtuoso to build an ADO.NET Data Services based application

Note: When working with external or 3rd party databases, simply use the Virtuoso Conductor to link the external data source into Virtuoso. Once linked, the remote tables will simply be treated as though they are native Virtuoso tables leaving the virtual database engine to handle the rest. This is similar to the role the Microsoft JET engine played in the early days of ODBC, so if you've ever linked an ODBC data source into Microsoft Access, you are ready to do the same using Virtuoso.

Related

• Entity Oriented Data Access
• Yoda & the Data FORCE.

After reading Bengee's interview with CrunchBase, I decided to knock up a quick interview remix as part of my usual attempt to add to the developing discourse.

CrunchBase: When we released the CrunchBase API, you were one of the first developers to step up and quickly released a CrunchBase Sponger Cartridge. Can you explain what a CrunchBase Sponger Cartridge is?

Me: A Sponger Cartridge is a data access driver for Web Resources that plugs into our Virtuoso Universal Server (DBMS and Linked Data Web Server combo amongst other things). It uses the internal structure of a resource and/or a web service associated with a resource, to materialize an RDF based Linked Data graph that essentially describes the resource via its properties (Attributes & Relationships).

CrunchBase: And what inspired you to create it?

Me: Bengee built a new space with your data, and we've built a space on the fly from your data which still resides in your domain. Either solution extols the virtues of Linked Data i.e. the ability to explore relationships across data items with high degrees of serendipity (also colloquially known as: following-your-nose pattern in Semantic Web circles).

Bengee posted a notice to the Linking Open Data Community's public mailing list announcing his effort. Bearing in mind the fact that we've been using middleware to mesh the realms of Web 2.0 and the Linked Data Web for a while, it was a no-brainer to knock something up based on the conceptual similarities between Wikicompany and CrunchBase. In a sense, a quadrant of orthogonality is what immediately came to mind re. Wikicompany, CrunchBase, Bengee's RDFization efforts, and ours.

Bengee created an RDF based Linked Data warehouse based on the data exposed by your API, which is exposed via the Semantic CrunchBase data space. In our case we've taken the "RDFization on the fly" approach which produces a transient Linked Data View of the CrunchBase data exposed by your APIs. Our approach is in line with our world view: all resources on the Web are data sources, and the Linked Data Web is about incorporating HTTP into the naming scheme of these data sources so that the conventional URL based hyperlinking mechanism can be used to access a structured description of a resource, which is then transmitted using a range negotiable representation formats. In addition, based on the fact that we house and publish a lot of Linked Data on the Web (e.g. DBpedia, PingTheSemanticWeb, and others), we've also automatically meshed Crunchbase data with related data in DBpedia and Wikicompany data.

CrunchBase: Do you know of any apps that are using CrunchBase Cartridge to enhance their functionality?

Me: Yes, the OpenLink Data Explorer which provides CrunchBase site visitors with the option to explore the Linked Data in the CrunchBase data space. It also allows them to "Mesh" (rather than "Mash") CrunchBase data with other Linked Data sources on the Web without writing a single line of code.

CrunchBase: You have been immersed in the Semantic Web movement for a while now. How did you first get interested in the Semantic Web?

Me: We saw the Semantic Web as a vehicle for standardizing conceptual views of heterogeneous data sources via context lenses (URIs). In 1998 as part of our strategy to expand our business beyond the development and deployment of ODBC, JDBC, and OLE-DB data providers, we decided to build a Virtual Database Engine (see: Virtuoso History), and in doing so we sought a standards based mechanism for the conceptual output of the data virtualization effort. As of the time of the seminal unveiling of the Semantic Web in 1998 we were clear about two things, in relation to the effects of the Web and Internet data management infrastructure inflections: 1) Existing DBMS technology had reached it limits 2) Web Servers would ultimately hit their functional limits. These fundamental realities compelled us to develop Virtuoso with an eye to leveraging the Semantic Web as a vehicle from completing its technical roadmap.

CrunchBase: Can you put into layman’s terms exactly what RDF and SPARQL are and why they are important? Do they only matter for developers or will they extend past developers at some point and be used by website visitors as well?

Me: RDF (Resource Description Framework) is a Graph based Data Model that facilitates resource description using the Subject, Predicate, and Object principle. Associated with the core data model, as part of the overall framework, are a number of markup languages for expressing your descriptions (just as you express presentation markup semantics in HTML or document structure semantics in XML) that include: RDFa (simple extension of HTML markup for embedding descriptions of things in a page), N3 (a human friendly markup for describing resources), RDF/XML (a machine friendly markup for describing resources).

SPARQL is the query language associated with the RDF Data Model, just as SQL is a query language associated with the Relational Database Model. Thus, when you have RDF based structured and linked data on the Web, you can query against Web using SPARQL just as you would against an Oracle/SQL Server/DB2/Informix/Ingres/MySQL/etc.. DBMS using SQL. That's it in a nutshell.

CrunchBase: On your website you wrote that “RDF and SPARQL as productivity boosters in everyday web development”. Can you elaborate on why you believe that to be true?

Me: I think the ability to discern a formal description of anything via its discrete properties is of immense value re. productivity, especially when the capability in question results in a graph of Linked Data that isn't confined to a specific host operating system, database engine, application or service, programming language, or development framework. RDF Linked Data is about infrastructure for the true materialization of the "Information at Your Fingertips" vision of yore. Even though it's taken the emergence of RDF Linked Data to make the aforementioned vision tractable, the comprehension of the vision's intrinsic value have been clear for a very long time. Most organizations and/or individuals are quite familiar with the adage: Knowledge is Power, well there isn't any knowledge without accessible Information, and there isn't any accessible Information without accessible Data. The Web has always be grounded in accessibility to data (albeit via compound container documents called Web Pages).

Bottom line, RDF based Linked Data is about Open Data access by reference using URIs (HTTP based Entity IDs / Data Object IDs / Data Source Names), and as I said earlier, the intrinsic value is pretty obvious bearing in mind the costs associated with integrating disparate and heterogeneous data sources -- across intranets, extranets, and the Internet.

CrunchBase: In his definition of Web 3.0, Nova Spivack proposes that the Semantic Web, or Semantic Web technologies, will be force behind much of the innovation that will occur during Web 3.0. Do you agree with Nova Spivack? What role, if any, do you feel the Semantic Web will play in Web 3.0?

Me: I agree with Nova. But I see Web 3.0 as a phase within the Semantic Web innovation continuum. Web 3.0 exists because Web 2.0 exists. Both of these Web versions express usage and technology focus patterns. Web 2.0 is about the use of Open Source technologies to fashion Web Services that are ultimately used to drive proprietary Software as Service (SaaS) style solutions. Web 3.0 is about the use of "Smart Data Access" to fashion a new generation of Linked Data aware Web Services and solutions that exploit the federated nature of the Web to maximum effect; proprietary branding will simply be conveyed via quality of data (cleanliness, context fidelity, and comprehension of privacy) exposed by URIs.

Here are some examples of the CrunchBase Linked Data Space, as projected via our CruncBase Sponger Cartridge:

1. Amazon.com
2. Microsoft
3. Google
4. Apple

In 2006, I stumbled across Jason Kolb (online) via a 4-part series of posts titled: Reinventing the Internet. At the time, I realized that Jason was postulating about what is popularly known today as "Data Portability", so I made contact with him (blogosphere style) via a post of my own titled: Data Spaces, Internet Reinvention, and the Semantic Web. Naturally, I tried to unveil to Jason the connection between his vision and the essence of the Semantic Web. Of course, he was skeptical :-)

Jason recently moved to Massachusetts which lead to me pinging him about our earlier blogosphere encounter and the emergence of a Data Portability Community. I also informed him about the fact that TimBL, myself, and a number of other Semantic Web technology enthusiasts, frequently meet on the 2nd Tuesday of each month at the MIT hosted Cambridge Semantic Web Gatherings, to discuss, demonstrate, debate all aspects of the Semantic Web. Luckily (for both of us), Jason attended the last event, and we got to meet each other in person.

Following our face to face meeting in Cambridge, a number of follow-on conversations ensued covering, Linked Data and practical applications of the Semantic Web vision. Jason writes about our exchanges a recent post titled: The Semantic Web. His passion for Data Portability enabled me to use OpenID and FOAF integration to connect the Semantic Web and Data Portability via the Linked Data concept.

During our conversations, Jason also eluded to the fact that he had already encountered OpenLink Software while working with our ODBC Drivers (part of or UDA product family) for IBM Informix (Single-Tier or Multi-Tier Editions) a few years ago (interesting random connection).

As I've stated in the past, I've always felt that the Semantic Web vision will materialize by way of a global epiphany. The count down to this inevitable event started at the birth of the blogosphere, ironically. And accelerated more recently, through the emergence of Web 2.0 and Social Networking, even more ironically :-)

The blogosphere started the process of Data Space coalescence via RSS/Atom based semi-strucutured data enclaves, Web 2.0 RDFpropagated Web Service usage en route to creating service provider controlled, data and information silosRDF, Social NetworkingRDF brought attention to the fact that User Generated Data wasn't actually owned or controlled by the Data Creators etc.

The emergence of "Data Portability" has created a palatable moniker for a clearly defined, and slightly easier to understand, problem: the meshing of Data and Identity in cyberspace i.e. individual points of presence in cyberspace, in the form of "Personal Data Spaces in the Clouds" (think: doing really powerful stuff with .name domains). In a sense, this is the critical inflection point between the document centric "Web of Linked Documents" and the data centric "Web or Linked Data". There is absolutely no other way solve this problem in a manner that alleviates the imminent challenges presented by information overload -- resulting from the exponential growth of user generated data across the Internet and enterprise Intranets.

A very cool video promo for Ruby on Rails and Mac OS X, or should I say: 37 Signals & Apple :-) Either way, very cool!

BTW - We have just released a collection of High-Performance Data Providers for ActiveRecord. Our providers deliver

Consistent Functionality

to RoR developers across Virtuoso, Oracle, SQL Server, Sybase, DB2, Ingres, Informix, and others without compromising performance or cross platform portability.

A very cool video promo for Ruby on Rails and Mac OS X, or should I say: 37 Signals & Apple :-) Either way, very cool!

BTW - We have just released a collection of High-Performance Data Providers for ActiveRecord. Our providers deliver

Consistent Functionality

to RoR developers across Virtuoso, Oracle, SQL Server, Sybase, DB2, Ingres, Informix, and others without compromising performance or cross platform portability.

(Cut & Pasted near-verbatim from Orri Erling's Weblog).

Recent Virtuoso Developments:

We have been extensively working on virtual database refinements. There are many SQL cost model adjustments to better model distributed queries and we now support direct access to Oracle and Informix statistics system tables. Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps Virtuoso optimize distributed queries. Also the documentation is updated as concerns these, with a new section on distributed query optimization.

On the applications side, we have been keeping up with the SIOC RDF ontology developments. All ODS applications now make their data available as SIOC graphs for download and SPARQL query access.

What is most exciting however is our advance in mapping relational data into RDF. We now have a mapping language that makes arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable. We will put out a white paper on this in a few days.

Also we have some innovations in mind for optimizing the physical storage of RDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We are experimenting with a new more space efficient index structure for better working set behavior. Next week will yield the first results.

(Cut & Pasted near-verbatim from Orri Erling's Weblog).

Recent Virtuoso Developments:

We have been extensively working on virtual database refinements. There are many SQL cost model adjustments to better model distributed queries and we now support direct access to Oracle and Informix statistics system tables. Thus, when you attach a table from one or the other, you automatically getup to date statistics. This helps Virtuoso optimize distributed queries. Also the documentation is updated as concerns these, with a new section on distributed query optimization.

On the applications side, we have been keeping up with the SIOC RDF ontology developments. All ODS applications now make their data available as SIOC graphs for download and SPARQL query access.

What is most exciting however is our advance in mapping relational data into RDF. We now have a mapping language that makes arbitrary legacy data in Virtuoso or elsewhere in the relational world RDF queriable. We will put out a white paper on this in a few days.

Also we have some innovations in mind for optimizing the physical storage of RDF triples. We keep experimenting, now with our sights set to the highend of triple storage, towards billion triple data sets. We are experimenting with a new more space efficient index structure for better working set behavior. Next week will yield the first results.

A nice piece of DBMS history. I certainly believe that DBMS market history is getting more relevant by the second :-) Enjoy the post!

(From Dave Kellog, Mark Logic's CEO)

Ingres: Can You Ever Go Back?: "In an eerie turn of events, my ex-, ex-, ex-employer Ingres Corporation has been resurrected by, of all people, Terry Garnett, one of Oracle's early marketing vice presidents, now of Garnett & Helfrich Capital. They have bought Ingres back from Computer Associates, open sourced it, and are building a RedHat-like, open-source business around it. To boot, they have built quite an executive team, including the recent poaching of well-respected vice president Bill Maimone from Oracle.

The whole episode reminds me of my favorite bad sci-fi movie, Escape from New York, where lead character Snake Plissken is consistently greeted with: 'Snake Plissken? I thought you were dead.' The same could be said of Ingres.

For those not familiar with RDBMS history, Ingres was one of the first relational database management systems (RDBMSs) and was created at UC Berkeley. I worked with Ingres at the Center for Computational Seismology at Lawrence Berkeley Lab while I was in school. (We let them use the tape drive on our VAX 11/780 and were given a free license in return.)

After graduating, I went to work for the vendor, Relational Technology, Inc., then run by CEO Gary Morgenthaler with brilliant visionary Michael Stonebraker acting as de facto CTO. When I joined Ingres in 1985, it was one of the 'big three' relational vendors.

• Relational Software, Inc., makers of Oracle, founded by Larry Ellison and Bob Miner
  
• Relational Technology, Inc., makers of Ingres, founded by Michael Stonebraker, Eugene Wong, Larry Rowe, and (I think) Jon Nackerud and Gary Morgenthaler.
  
• Relational Database Systems, Inc., makers of Informix, founded by Roger Sippl.
  

Both Ingres and Oracle were approximately $30M in size in 1985. Informix was a bit smaller. (For those wondering 'where's Sybase?' they entered the market approximately 5 years after the big three.)

Ingres placed me in the bizarre situation of experiencing great success and great failure, simultaneously. On one hand, during my 7 years there we went from being a $30M company to a $250M division of a $400M company, and I went from first-line technical support rep to director of product marketing.

On the other hand, in the same timeframe, Oracle went from $30M to $1B, won the second largest opportunity of the 20th century (the first was PC operating systems), and left the broken 'People's Republic of Ingres' in its dust.

Others have written the Ingres epitaph. Here is my version. Ingres, in my estimation, failed for the following reasons.

At a product level:

• The wrong query language. Ingres bet on Quel. Oracle implemented SQL. While many (including the notable Chris Date) felt that Quel was 'better,' it didn't matter. IBM had stated its intention to implement SQL, making SQL a de facto standard. This was a huge difference and it's often forgotten. As late as 1990, Ingres was still selling a native Quel engine that preprocessed SQL to Quel on the front-end. Differing semantics between the languages and the echo-back of Quel from the server when SQL was sent to it, all sent smart customers running in the other direction.

• Page-level locking. Oracle had row-level locking. Ingres had page-level locking. Oracle effectively rammed this difference down Ingres's throat in virtually every sales situation. Later, Sybase would suffer a similar fate, particularly with applications vendors like SAP who refused to implement on Sybase until it had row-level locks.

• Lack of read consistency. The only way for readers to not block writers in Ingres was to set 'READLOCK = NOLOCK.' (This was about as poorly chosen a piece of syntax for marketing purposes as SET SERIALIZABLE = FALSE in early Oracle versions.) Oracle offered read-consistent snapshots, leveraging timestamps and the logging systems's before image file, that enabled a consistent view without blocking updates.

• Lack of connect by. Oracle added connect-by to SQL enabling the transitive closure of a table, most commonly needed in bills-of-materials and other 'parts explosion' type queries.

• Portability strategy. Oracle did a much better job of porting not only to more platforms, but keeping the product the same across them. Ingres attempted to optimize more for each platform (e.g., squeezing the product into 640K on the PC by dropping functionality) which, while perhaps counter-intuitive, was a mistake.
  

At a business level:

• Failure to understand the tornado. The tornado refers to Geoffrey Moore's metaphor for the hypergrowth phase of a high-tech, infrastructure market. During that phase, Moore argues that vendors should 'just ship' in an attempt to gain as much market share as possible so as to pop out of the tornado as the clear market leader. During the tornado, increasing returns happen -- the more clear your leadership, the more customers want to buy from you. The 3-5 year tornado determines who will lead the market for the next decade. Ingres missed this, was timid when it needed to be aggressive, and lost.
  

• Failure to understand the 'best product' doesn't necessarily win and that best product is defined in the mind of the customer. I joked that my job in 1989 was to explain why you didn't need row-level locking when you had a 2K page size, but that didn't matter. Customers wanted SQL, if inferior to Quel. Customers wanted row-locks, if somewhat unnecessary given a small page size. Customers wanted read consistency, which was indeed absolutely necessary. Product marketing literally begged R&D and the company for these features, but they were (1) deep architectural limitations and thus 'hard' to fix, (2) deemed somewhat unnecessary at a technical level by engineering, and (3) generally viewed as sales and marketing problems that should be sold-around.
  

• Failure to understand sales and marketing. The company generally didn't 'get' either sales or marketing, underinvested in both, and in marketing's case had a revolving door of executives of all ilks (e.g., engineers, alliances people, consumer marketers) except those who understood the products and the customers. I had something like 10 bosses in marketing in 4 or 5 years.

So every year, Oracle planned to double and Ingres planned to grow 50% or so. Every year the execs told us this was the year that Oracle would get its comeuppance. Every year, Oracle doubled, or more than doubled. Ever year, we found it harder and harder to make the 50% growth target.

The laws of compounding took effect and across some 7 years Ingres went from being 100% of Oracle's size to 25%. Oracle, indeed, hit the wall around 1990, but it was too late. Ingres had lost. So many people were so invested in Oracle that it literally couldn't fail. Larry Ellison got about $100M from the Japanese (NTT), restated revenues for all the bricks that had been shipped over the years (as I recall removing an entire Sybase from its books at the time), and turned the company around.

Ingres was bought by ASK in 1990 and sold to Computer Associates around 1992. I thought it would rest in peace in the CA cemetery for eternity, until I learned of its recent spin-out.

To say that Ingres had a strong corporate culture is an understatement. In fact, it lives on today at the Ex-Ingres website, complete with one of my favorite slogans: 'Ingres corporate culture without the corporation.'

Many successful companies sprung from Ingres. Documentum and Forte are two of the bigger successes. The Forte crowd lives on today at AmberPoint. John Newton, one of Documentum's two founders, is trying his luck at open source with Al Fresco. Lesser known but quite successful, Perforce, was founded by lab-coat-wearing Ingres engineer Christopher Seiwald who, after reading Positioning, learned that a company should try to own one word in the mind of a customer, decided to build the fast configuration management tool, and quite successfully did just that.

Will the new Ingres be successful? They have quite a team, but one never knows. The whole open source vs. subscription/ASP vs. traditional enterprise software licensing battle is in reality just beginning. How I think it all ends will be the subject of another post."