It’s day 16 of the 30 Days to Become a Better Monarch Modeler series and for today I had scheduled a discussion about combining data sources with Monarch.

But reviewing some of the existing content on the topic already available here on ExcelWithMonarch.com, most notably a somewhat lengthy comparison of the available internal and external lookup methods, leads me to think that I’ve already described a good portion of what you need to know about the technicalities.

So I thought it best, rather than somehow restating and repackaging it, to instead focus exclusively on how combining data sources can help you become a better Monarch modeler.

What’s The Purpose?

The main idea reason to combine data sources is that we need to add new data to the original data and probably use that as a basis for further analysis, probably employing this new data as a key field in a summary.

Most information systems offer some kind of sales report, but I haven’t seen many that have included different types of geographic data in that report. Perhaps you need to analyze the basic results by breaking down the basic country-wide east, west, north and south regional sales. Maybe it’s a lot closer to home, but your report just includes a handful of nearby city names, and you’d really like to isolate the various neighborhoods or districts within a city. In either case, you can accomplish this easily and with just a few clicks in Monarch.

The key, as it were, is to isolate either a single field or a combination of two or three fields that you can use to connect the dots between your data sets. What you want to do is find a way to create unique connections. Examine the source report and determine which field you could use to create a list of unique meaningful values. Customer and vendor numbers often work well, as do part numbers. Other times you might find category names and geographic names will do just fine.

Let’s stick to the geography concept. Suppose that using our customer master list that displays address data, we derive a list of unique US state names in which our customers operate. We develop a model for this report and export the customer numbers, customer names and state names to an Excel 2007 file. This is great format to use for lookups because Monarch reads them very quickly and you can store over a million records in your lookup table. That’s not a large table if it’s storing transactional data, but it is a lot of records for a simple lookup table of unique values.

Let’s say that we know from our past experience with the reports available to us, sometimes a particular report will display the customer number, sometimes the name, and sometimes both fields. Now with our exported list, we’re ready for any report than contains customer information.

Before we can begin using our exported file as an input source, we need to add some other useful data to it. We would open the file and add columns for Region and it values (North, East, South, West, Northeast, Northwest, Southeast and Southwest), and another field for the customer’s main business type or market segment (Retail, Wholesale, Manufacturing, Restaurant, Consumer Service, what have you) and maybe even a sub-type (Electronics, Food & Beverage, Automotive, etc.).

Now you have a list that can provide different insights than could gain prior to your developing this list. You can report your sales by Business Type and Sub-Type. But you can just as easily now incorporate this information into an Accounts Receivable report (which business types are having trouble paying your bills?) or you might convert your simple sales levels report into a profitability report by Business Type and Sub-Type (which industries are your most and least profitable?).

These are pretty simple little examples, but it’s the possibilities that are important. I’ve never seen a single system that delivers, as a canned or standard report, everything that management would like to have or would otherwise be a beneficial analytical tool.

Sometimes you might be lucky and you‘ll be supplied with files that you can immediately use as lookup sources. Other times you’ll need to develop them yourself.

In either case, the only thing holding you back as a Monarch modeler contributing to your organization is your imagination and perhaps your overall knowledge of your organization.

If you can quantify or otherwise describe something in a new way that doesn’t already exist in your system, then your Monarch skills will be of great value. If you can build useful tools – reports and other analyses – by employing lookups in your models, that haven’t even been envisioned by anyone else in your group, then you’ll be ahead of the curve and in all likelihood will be a real asset to the group.

Some Technicalities

Many external lookups can be created using a single common key field, but don’t forget that you may need to use multiple key fields to tighten up the connections a bit from time to time, as needed.

One of the common challenges encountered is the message indicating that “The Source and Table window columns must be of the same type. This is telling you that Monarch has detected the data type of the fields that you’ve listed as linking columns, and that they’re not the same type of field.

This is frequently encountered when a field that you believe contains only numeric values actually contains, somewhere in the list, an alpha character or a devilish space. That cell that you thought was empty might not be; it might contain one or more space characters.

Another common problem is encountered because of the way in which Monarch determines the field types in the remote Excel file. Monarch examines the first 250 records, and if they’re all, say, numeric, then it believes the entire column to be numeric. But if you have a product number of 1000A in there somewhere after record number 250 you’re about to run into trouble.

The easiest solution is to pre-sort your lookup file before attempting to connect to it with the Monarch model. Sort the Excel table so that any text values appear at the top of the list, and you’ll avoid many problems.

Unfortunately there’s not a easy solution if you’re linking to a text file, like csv. Using Access databases as a source should experience this problem as Access can better control the field content. Without a pre-programmed macro running behind the scenes, Excel doesn’t care what you put into a cell; anything will be accepted.

Your Task for Today

Today’s content has largely been conceptual in nature. If you haven’t already read the existing content on ExcelWithMonarch.com concerning lookups, please do so today; especially if you’re new to Monarch and haven’t used lookups at all in your modeling efforts to date.

In particular, I’d once again direct you to the comparison of internal and external lookups as there continue to be times when one approach is warranted over the other and being aware of the differences may affect how you choose to build your models.

Speaking of building models, some work to do in addition to the reading: today you should build both internal and external lookups into one of your models. Experiment with using different file types as inputs for your external lookups.

Did you realize that you can create multiple external lookup definitions? You can even use the retrieved values generated by one external lookup as the key field of another external lookup! Using lookups, you can tie a lot of information together and derive final data sets that you may have thought was nearly impossible to generate.

Come Together

The critical piece to using lookup fields in your models is dreaming up what you want the end result to include. What will the structure of your summary be? Which fields are readily available in your report, and which will you need to gather from elsewhere, and how will you connect the dots? What does the business need? What already exists in one manner or another? What can you contribute?

The Monarch work is easy, now that you know how. The hard part is up to you. But when you combine them it will be clear that you excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 17 of the series, or review Part 15.

Once you’ve built the templates and calculated fields required to extract all of the data within a given report with Monarch, do you save your model, close it, and carry on to the next project, or do you spend some additional time with the report to transform your model from being perfectly functional and workmanlike to being the high performance analysis tool that it can be?

Adding some filters to the model often lets you make sense of your data, revealing both the gems and the lumps of coal that are contained in your reports. To create truly powerful filters, you’ll want to master the Boolean logic (a fancy term with simple impacts) that Monarch offers with its operators.

Those familiar with filtering data with Excel know that you’re not only limited to data found on a single worksheet, but that defining and altering a filter that involves multiple fields is a clumsy exercise at best. On the other hand, the process of defining, changing and activating filters with Monarch is not only easier, but far faster.

Since multiple summaries can be built to be based on specific defined filters, you can not only create multiple views of a particular data set, but your filters can provide multiple data sets while you’re working with a single set of reports, or even just a single report.

Now that special, custom views of your data are available, you can export those and re-purpose them as inputs for one or more additional Monarch process too. Monarch Pro users can use the exported files as database sources for new inputs and/or data that can be used in other external lookups to provide even more analytical power to static reports.

By exporting summaries without totals, you can build what are essentially multiple data tables. Make sure that you do not suppress duplicate values for your key fields when building those summaries, though.

Shine on, you crazy diamond

By using more than just the regular data extraction tools when building Monarch models, you can flush out everything that is hiding in those plain Jane reports. Have a good look at what sort of content is in your next report and think about what you might be able to do with it that might be a little unconventional. You might just find that you can stop working so hard to find the treasures, and instead can have those treasures work for you.

When you provide the insights that allow for truly informed decision making – most especially when what you’re generating doesn’t already exist in another system – those on your team that count on you will know that you really do excel with Monarch.

In a prior post, I wrote about how Monarch outperformed Access as a report writer. Today the wind has blown the other way, and we’ll focus on how Access can provide additional functionality for your Monarch models.

Monarch user Diane recently asked whether a logarithmic function was available in the software. Unfortunately it is not, so the hunt was on to find an alternate solution.

Initially I thought that she might be able to export values out to Excel and then apply Log functions to the data, and would then finally be able to import back into Monarch with an external lookup. But the large data sets would prevent that with Excel 2003.

And that was a good thing, because it forced me to look for another option, and what I devised was, I believe, a better solution.

It occurred to me from my SQL programming efforts that many functions can be called in the process of querying the database, so I checked to see if the Log function could be called in an Access query. It could!

This small act can let you effectively add to Monarch’s abilities pretty easily. I described the steps to take in detail on Datawatch’s forum, but here is the general concept.

First, we need to create the initial Access database in which to build our custom query, so export the table records to a new Access database. For this type of need, there’s a good chance the query will be used regularly, so be sure to setup a Project Export with Monarch.

There’s an important step while defining the export, and that is to tell Monarch not to overwrite the file when exporting, but instead to add data to the database. Why is that? If Monarch overwrites the Access database, it will destroy the custom query that will be defined in the next major step. Finish defining the database definition by electing to overwrite the destination table within the database and running the export.

Now start up Access and open the newly created database and create a new query with the query wizard. Add all of the fields that you need to the query and select the default of Detail and finish the query. Open your new query and under the View menu select SQL View. You’ll see something like

SELECT Table.Field1, Table.Field2
FROM Table;

This is where you can define a new field using a function. Just add a little bit to the SQL that Access creates.

SELECT Table.Field1, Table.Field2, Log(Table.Field1)/Log(10) AS LogValue
FROM Table;

The “AS” lets you name the field as you desire, instead of letting Access assign a name for you.

Saving the query stores your customization in the database and this addition to the database will be used by Monarch Pro. This is the next step of the process.

Back in the Monarch model, create a new external lookup. Select the Access database, and instead of selecting the table that was exported from Monarch, select the new query.

When Monarch runs the query, the SQL statement calculates the function using the values in the table, and the results of those calculations are brought into Monarch.

OK, interesting enough, but what can be done with it?

There are a number of functions that are available in Access that aren’t built into Monarch. Which one can you use to your advantage to help you excel with Monarch?

One of Monarch’s greatest strengths is its ability to add even more related data to the new data that has just been extracted.

This gives all kinds of flexibility and analytical capabilities to your main information system, be it a small accounting package or a large ERP system, that it doesn’t currently provide. Imagine being able to very quickly get the results that having your database modified and populated – complete with new reports – would provide. Now imagine achieving those results yourself. That’s the power that Monarch provides.

The two versions of Monarch, Standard and Pro, each offer an internal ‘lookup’ style calculated field, while the Pro version adds an external lookup so that you can import values from sources outside of your model automatically.

There are compelling reasons to use, and avoid, both types of lookup field. Let’s take a moment to examine them both, shall we?

While we’re not really comparing the Pro and Standard editions of Monarch (I’ll save both you and me another extended post here: just get the Pro version and thank me later ), an advantage of Version 9 Pro versus the Standard edition is its ability to import objects (like lookup fields) from existing models.

How does this relate to comparing internal and external lookups?

With internal and external lookups previously defined in existing models, you can make quick work of implementing lookups in a new model by simply importing the previously defined lookup into your new model. This is especially beneficial when the original is an internal lookup with a long list of output values. Build it once and reuse it as and when you need it. You can even define the import to be linked to the original lookup so that when the original is manually updated those edits automatically carry over to your other models.

New and Improved Analytical Abilities for Your Organization!

I can’t overemphasize the impact that using this single Monarch feature can have on the value of your work. It’s the first step towards developing new analytics and new insight for your organization.

By constantly looking for meaningful ways to add value to your data with lookups you’ll definitely excel with Monarch.

This past weekend, it’s possible that a number of Excel users were introduced to Monarch for the first time. As they may not have even heard of the product before, I thought that this might be an opportune time to demonstrate how one can work with both Excel and Monarch simultaneously.Today’s video shows how to add to values from Excel to report data that you’ve extracted with Monarch. We’ll use Monarch’s external lookup feature to add some values (which we’ll use in new calculations) to the model we created in a previous video post.

While the data set in the video is small, you can see how you can have Monarch effectively use Excel files for both inputs and outputs.

Without custom programming or complex formulas in Excel, you can very quickly add new data elements to the report files generated by your existing systems.

If you haven’t seen the first video, in which the process of essentially teaching Monarch how to extract data from a specific report is demonstrated, view it now.

Today’s video extends the initial model by adding the external lookups and a calculated field. (View the external lookup demo video now.)

Monarch’s engine imports data from Excel workbooks far more quickly that even Excel does with its VLOOKUP function. Even if you’re accustomed to using Excel’s lookup functions, you’ll gain considerable performance advantages by having Monarch connect your data if you’re working with any nontrivial amounts of data.

Becoming adept at developing new decision support tools which easily connect related data is a great way to excel with Monarch.

Hook up with Lookups!

One of the best reasons to use Monarch Pro is its ability to easily add more data to what you extract from your initial data source. This will allow you to analyze your data in ways not possible with your existing source.

The Problem

Let’s say you have a sales report that shows units sold by customer, but you want units sold by sales rep, and all of your available reports only give you dollar amounts based on sales rep, not units. Further, the sales rep names don’t appear on any of the customer based reports. What do you do?

Typically, you’d first get the customer sales data into Excel. Then you’d build a separate list showing the sales rep names for all of your customers, probably including that as another sheet in file. Finally, you’d probably use one of Excel’s lookup functions to get the rep names into your unit data, based on the customer name.

If you’ve got up to a few hundred rows of unit data, and a handful of customers, that’ll be OK. But have you ever used lookup formulas across thousands, or tens of thousands of rows? I hope you’ve got a really powerful PC and/or a lot of time on your hands, because every time you recalculate that spreadsheet it’s going to take awhile. If you didn’t include your rep list within your file, and it’s in a different Excel file located on a network drive, you may as well go for lunch now. I hope you don’t have a tight deadline for this!

And that all of that assumes that you’re well acquainted with the lookup functions and all of their particular intricacies.

The Solution

Instead, use Monarch Pro’s external lookups. You still need the list that tells you which rep is for which customer, but you’ll be done in a fraction of the time, and your final data set in an Excel file will be much smaller, calculate faster, and will be easier to use and share with less experienced Excel users as it won’t have all of those complex functions.

With Monarch’s external lookups you can easily add a number of related fields to your data, and the end user of your final data will have a complete data set without needing detailed knowledge of specific Excel functions and features, or access to the potentially remote data source.

External lookups can easily bring a lot more data into a very simple report, producing a very powerful decision making tool and helping you excel with Monarch.