The ability of Monarch to sort key fields in both the Table and Summary windows is tremendously useful as it enables us to further analyze our data. The ability to focus only on the most important data can be critical when working with data sets of any size.

photo credit: sam_churchill

Let’s assume that we have a list of 36 records reflecting banking transactions. We want to create a list showing only the largest value each transaction at each of the branches. In our data set, there are 12 transactions at each of three branches. Once the entire data is filtered, we expect to see only three records: one for each branch number, with the largest transaction amount that occurred for that branch displayed.

Monarch makes it easy to sort records on multiple key fields. In fact, within a single Monarch model, one can define multiple manners in which the data will be sorted at any given time as is demonstrated with the Sort Orders dialog box.

Users of Monarch v10 and later have an advantage over users with earlier Monarch versions when filtering data that employs a defined sort order, as they can build filters that take the active sort order into consideration.

By utilizing the Advanced properties of the filter definition, we can specify that we need to handle duplicate values for a particular field carefully.

By activating first the custom sort order and then the new filter Monarch will list only three records in our sample data set, as expected.

Notice that the option for duplicated rows reads “first row only, as sorted”. That means that we’re effectively restricted to only a single row when duplicates are encountered, as is the case with our 12 records per branch.

Note that we’ve selected the “Unique Rows” option in addition to the “Duplicated rows” option. By using this option we ensure that we always have a representative record for every branch location. This would be necessary if in our set of 36 records we had a distribution of 20 records for one branch, 15 records in another and only one record for the final branch.

Top N Records

What would we do if we wanted not just a summary of our data but rather a complete list of all of the fields available for the top, say, 10 records for a given key field? Monarch doesn’t make it easy (in that an option display such data readily doesn’t exist) but for Monarch Pro users it is possible.

The goal of this challenge is to filter the Table window to display only a particular number of records for every key item value. Employing summaries are effective for drill-up and drill-down solutions, but not for listing the top n detail records for all key values. We’ll need to build a filter in the Table window to satisfy this requirement.

First, build and apply the custom sort order as above. Next, add a new numeric calculated field named SortRowNo using only the expression Rowno(). We’ll use this field as a key part of the solution. Astute readers will be quick to point out that filters cannot be built when any component of the filter employs the Rowno() function. Monarch will complain loudly.

Instead, assuming that this is a process that would be repeated regularly, you’d define a project export to export the unfiltered Table window to a temporary file (Excel workbook, Access database table, or what have you, as applicable). Save the model and project files, and run the project export.

Now in a new Monarch session, open the freshly exported file as a database source. Import all of the fields. Now that the SortRowNo contains only values, and not the Rowno() function, we can use those values in a filter. But we need one more value before we create that filter.

What we’ll do next is a little odd. We’re going to create a new external lookup. The data source for the lookup will be exactly the same file that we just opened, and the key field will be, in our case, the branch identifier. Monarch will complain about the specified field not forming a unique key to the external table. That’s OK. Click the “Yes” button and continue to import the SortRowNo field as LookupRowNo.

When the external lookup is performed, all equivalent branch identifiers will be assigned the same value for LookupRowNo: the row of the Table of the first model in which that branch first appeared.

Now with a complete data set, we can define a filter which will result in the top 10 records that we’re after. For the filter use the expression:

LookupRowNo – SortRowNo < 10

Build in Flexibility

By replacing the fixed value of 10 in the filter formula with a new runtime parameter calculated field (named rtTopN for example), we can make the model much more user-friendly and flexible. Upon opening the model, Monarch will prompt the user for a value to be used in the filter.

Additionally, values for runtime parameters can be passed to a model via a batch file to make the process even smoother. Imagine a business process that monitored the top 10 records each week, but only the top three items were required for monthly reporting. A batch file could be created and copied with only a slight edit in the copy to change the value for the rtTopN amount. Use the /set command line parameter to assign a value to a runtime parameter calculated field.

Ignore the Restrictions

This approach demonstrates that we can sometimes circumvent the technical restrictions that Monarch imposes to build solutions that resolve the challenges that exist in your information systems, whether they’re large or small.

Earn a spot in your organization’s Top 10 list. Excel with Monarch.

Well here we are again with another unexpected interruption of the 30 Days to Become a Better Monarch Modeler series. This time around, I had to take an unplanned business trip that threw a bit of a proverbial wrench in my machine. This was followed by returning to a sick family, and eventually and inevitably, becoming ill myself. Then a power failure during a stormy day managed to fry the external hard drive on which I stored the next couple of series posts which I’d prepared in advance despite the (now clearly inadequate) power protection system.

Regardless, we’ll still have 30 “episodes”; it’ll just take a little longer than planned to get through them. Sorry folks.

Any time that we work with raw data, more often than not we’re not really interested in all of the data that we have access to in this particular set, and it’s difficult to see the forest for the trees, as it were.

To that end, Monarch provides a fantastically rich ability to isolate important data through a mechanism that it calls Filters.

On day 12 we’ll introduce this vital feature to the 30 Days to Become a Better Monarch modeler series.

Filtering in Monarch is similar to using a search engine to find things of interest on the Internet. Usually it’s true that the better phrased the search, the better the results returned by the engine. Monarch acts the same way. Think of filters as Monarch’s way of searching for specific data within a larger set.

When do we want to employ Monarch’s filters? They are most beneficial when:

• The original data set is very large, and we’re only really interested in a particular subset for the task at hand. An example might be a report that’s been run for company-wide data, but only data for a department or two is currently required.
• When we need to focus very exactly and narrowly within the available data. Maybe we need to see the products shipped to southern states in the 15 days prior to the end of the quarter that generated greater less than 15% margins.
• When we cannot avoid bringing certain data into the set due to the methods used in developing the extraction templates. If it was not possible to avoid bringing the subtotal values that are interspersed throughout the report as the vendor changed into the data because there just wasn’t an apparent method to build the template without capturing those values, then they can be removed temporarily from the data by filtering out those subtotals.

Formula-based filters

In the end, there really only are three approaches to building filters:

1. To define the rules so as to include specific data,
2. To define the rules so as to exclude specific data,
3. To define the rules so as to include one aspect of the data and exclude another.

What’s the right and most efficient approach to take? It depends.

But no matter which approach is used, it will help you as a Monarch modeler to be entirely familiar with each of the available logical operators, and as your filter expressions grow in complexity, the order of operations and Boolean logic. Especially useful for filtering are the .In. and .NotIn. operators.

Monarch’s functions can play a huge role in building filter expressions, and allow for more flexibility than can possibly be demonstrated here, but it is worth mentioning that the RowNo() function cannot be used in filter expressions.

At times the filtering criteria can become quite complex; more than a few conditions are required to be met to generate the appropriate data. It’s then when Monarch’s ability to define multiple filters, and to combine specific individual filter, that make Monarch shine when compared to other software. Even Excel doesn’t come close to Monarch ease of use in this regard.

Monarch offers the ability to define a filter that is made up of other filters. Monarch calls these compound filters. In a prior paragraph there’s an example of filtering sales by geography by date by margin. To make the possible complexity more manageable, it’s often best to break a filter such as this down to the basic building blocks.

To do this, build three “formula-based” filters. First, create a filter than returns only the sales that occurred in the proper geography. Name this filter “Southern Sales”. Next, create another filter that returns all sales within a certain date range. Name this filter “Quarter End Sales”. Finally, create a filter that returns low margin sales (below 15%), and name this filter “Low Margin Sales”.

Compound Filters

To create the final filter, create a new filter but this time select the “Compound” option instead of “Formula-based”. In the Components tab of the dialog, assign a meaningful name to the filter, such as “South Low Margin End Qtr Sales”. You only have 31 characters for filter names, so name them wisely. Click on the items in list of available filters. You can Shift-click to select a range of filters, or Ctrl-click to select a collection of individual filters. Next, click the Add button to include the selected filters in the new filter. Notice that a compound filter expression is displayed in the bottom of the dialog. Our example would read [Southern Sales] AND [Quarter End Sales] AND [Low Margin Sales].

Notice that the option button is set to select rows that satisfy all of the component filters. Click the other option, “any of the component filters”. What does the filter expression read now? Each AND changed to OR. What would the effect be on the results returned by the filter?

What if we wanted to customize the filter a bit more, say to have the formula revised to read [Southern Sales] AND [Quarter End Sales] OR [Low Margin Sales]? Click the formula and revise the second AND to be OR. Well, that’s frustrating! Monarch won’t allow direct editing of the formula.

The solution is to save the first part, [Southern Sales] AND [Quarter End Sales], as a compound filter (South Quarter End), and then create a new compound so that the formula becomes [South Quarter End] OR [Low Margin Sales]. Using this technique we can define any filter imaginable.

Value-based filters

New to Monarch v10 is the ability to bring a list of values for a particular field into the filtering mechanism. The end result is the same as it would if a formula based filter had been created using only the .In. or the .NotIn. operator, but it’s much easier to use as the various values don’t need to be keyed into a formula. Instead, the values from the current data set are brought into the filter dialog just as when building internal lookups. The difference here is that a column of checkboxes are available. To include a particular value in the filter, just click its “Select” box.

As well, existing filters can be employed to limit the values that brought into the list. If it were up to me, I’d probably have placed the dropdown list of the filter names on the left side of the “Add current field values” button. As it is on the right side, it doesn’t tend to visually lead one to think that choosing a filter will affect the results of clicking the button, like it might if it were on the other side. Still, this is pretty minor and the functionality is terrific.

Your Task for Today

Monarch’s ability to isolate specifics from a vast sea of information is one of the first steps in the act of transforming data into information. It’s probably a fair statement to say that most of us just cannot mentally process huge amounts of information, and we find it much easier to decipher “bite-sized” chunks.

Now that you have a foundation in the filtering tools available, today you should create at least three filters using each of formula-based and compound mechanisms, and three more using value-based if you have Monarch v10. Mix it up and experiment with different functions for the formula-based filters. Before you finish build one more formula-based filter that duplicates the function of one of your compound filters, just so you can appreciate just how much easier it is to work with compound filters.

Don’t Strain Yourself

There’s no point in making your data and information tasks more difficult than they need be.
Not that long ago I had a coworker who, every Monday morning, would print a lengthy report and would meticulously review the stack of paper with a ruler and three highlighters in hand, essentially filtering his data. What a waste of time, effort and resources.

Stop working for your data. Make your data work for you. Filters are easy to use, easy to manage, and make it easy to excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 13 of the series, or review Part 11.

Once you’ve built the templates and calculated fields required to extract all of the data within a given report with Monarch, do you save your model, close it, and carry on to the next project, or do you spend some additional time with the report to transform your model from being perfectly functional and workmanlike to being the high performance analysis tool that it can be?

Adding some filters to the model often lets you make sense of your data, revealing both the gems and the lumps of coal that are contained in your reports. To create truly powerful filters, you’ll want to master the Boolean logic (a fancy term with simple impacts) that Monarch offers with its operators.

Those familiar with filtering data with Excel know that you’re not only limited to data found on a single worksheet, but that defining and altering a filter that involves multiple fields is a clumsy exercise at best. On the other hand, the process of defining, changing and activating filters with Monarch is not only easier, but far faster.

Since multiple summaries can be built to be based on specific defined filters, you can not only create multiple views of a particular data set, but your filters can provide multiple data sets while you’re working with a single set of reports, or even just a single report.

Now that special, custom views of your data are available, you can export those and re-purpose them as inputs for one or more additional Monarch process too. Monarch Pro users can use the exported files as database sources for new inputs and/or data that can be used in other external lookups to provide even more analytical power to static reports.

By exporting summaries without totals, you can build what are essentially multiple data tables. Make sure that you do not suppress duplicate values for your key fields when building those summaries, though.

Shine on, you crazy diamond

By using more than just the regular data extraction tools when building Monarch models, you can flush out everything that is hiding in those plain Jane reports. Have a good look at what sort of content is in your next report and think about what you might be able to do with it that might be a little unconventional. You might just find that you can stop working so hard to find the treasures, and instead can have those treasures work for you.

When you provide the insights that allow for truly informed decision making – most especially when what you’re generating doesn’t already exist in another system – those on your team that count on you will know that you really do excel with Monarch.

It’s usually around this time of year that the panic starts in offices everywhere.

It often begins with “We need a clean list of contact information for the annual mailing”, and is just as often followed by an inordinate amount of stress on the part of the person who’s responsible for compiling the list.

The trouble spot, and the staffer in question knows this all too well, is that even though every organization has some kind of contact management system – maybe stand-alone, maybe integrated into something else – someone along the line has made data entry mistakes that mess up the list and what ought to be a pretty straightforward exercise. There are city names in the state field, and all manner of other chaos waiting to be discovered.

All of this, of course, will make filtering or otherwise reviewing the list more difficult than it ought to be, and may take more time and effort to generate a usable list than is available.

Monarch Finds the Problem Areas for You

Monarch offers a really unique feature for handling and isolating address information. The Address Block feature converts either a multi-line character field or several unique fields and automatically determines the logical pieces and splits them into distinct named fields.

First, you tell Monarch what types of address styles to look for in your report: US, Canadian, UK, etc. Next, select the field or fields that contain the address information you need. Finally, you select the fields that you want Monarch to generate from your initial data.

There’s also an option to generate an error code field, and including this field is always a good idea. If Monarch thinks that the address generation process went well for a particular record, it will give the error code field a zero value. But if Monarch believes that something doesn’t look right, it will supply a value of one through five. The descriptions for those values can be found in Monarch’s online help.

Using the error code field is a great way to screen your data, and you’ll find problems with the addresses in the report very quickly.

And if you do find errors, you can make corrections within Monarch. Just create a user edited style calculated field or two if you really must make some manual tweaks, or if some data isn’t just right, build another Address Block with the fields that you list re-sequenced in a way that you believe will let Monarch give you better results.

Once the list seems pretty accurate, you can use Monarch’s filtering ability to isolate particular regions, customer names that begin with the letter M, or even those within a particular zip code.

Directions for Useful Addresses

When all of that is done, it’s a simple affair to export the data out of Monarch to an Excel file or an Access database so that it can be used as an easy data source for a Microsoft Word document for a mail merge, providing personalized letters or just plain old mailing labels. After all, old fashioned snail mail isn’t totally dead yet.

You can find your troublesome address data automatically, stop working so hard to get the data right, and have your data do a good portion of the work for you. All you need to do is excel with Monarch.

When I’m at my desk for anything longer than just a couple of minutes, you can almost guarantee that there will be music playing. And it’s usually one of the many songs I copied from my constantly growing CD collection to my computer. It’s no wonder that the iPod and portable media players in general have taken off as they have. Using electronic copies is so much more convenient.

But when your collection grows it can make it quite to difficult to locate something specific. Unless you have a photographic memory, and if you do: lucky you! So when you hear only a part of song on the radio that you haven’t heard for awhile, and want to listen to the whole thing, how do you find it in your collection, if in fact you even have it?

At this point I should mention that I’m a bit of a renegade, and don’t have an iPod so I don’t use iTunes. I use an old freeware media player that doesn’t have fancy library functions like most of the new stuff does and I’m OK with that. My favorite player software works just fine for me in every other way, so I leave well enough alone.

Well you could start going through your cabinet, and if it’s anything like mine, you might find it eventually… Or you put Monarch to work for you!

It’s More Like Monarch for Fine Tunes

Start with creating a text list of your files (songs). This starts out very retro, in today’s Windows world. Go to the DOS prompt (Start, run command, or cmd). Switch over to the drive that has your songs, if necessary, and enter:

dir /s > myfiles.prn

When this completes, fire up Monarch and open myfiles.prn.

DOS produces a nice Monarch-friendly list, but depending on how you organize your files you may have song titles appearing in groups, separated by folder name. I use folders named by artist, then subfolders for album title, with only the songs for that album in that folder. It’s some sort of obsessive compulsive thing, I know. Oh well, a detail-oriented personality, right?

This sort of thing is right up Monarch’s alley. All of the individual file names become the detail lines (trap on the date, and specify a couple of blank traps where the <D characters appear on other lines in positions 25 and 26. Paint your field for the file names beginning in position 40. I went quite wide – 165 characters – to accommodate my longest file name.

Now add the folder as an append. I trapped using “directory of”, and painted the field from position 17 and went 127 wide.

That’s the heavy lifting done. Now for a little finesse. In the Table window, create a couple of calculated fields. Given that I use a fixed folder structure, I used this formula to extract the Artist name:

LSplit([ParentFolder],5,"\",4)

Then I used this similar formula to extract the Album name:

LSplit([ParentFolder],7,"\",5)

Then I took a couple of minutes to build summaries using Artist, Album and File Name as key fields, using count as the measure. This results in a nice, easy to read inventory report.

Finally, adding a runtime parameter field (SearchFor), that in turn is used in a filter formula (and saved as a separate model file), lets me look for songs that contain a certain word in the name:

Instr(SearchFor,[File Name])>0

All Play and No Work?

This whole exercise really only took a few minutes, and gave me a great tool for searching and managing my files. And while it was all in the name of fun, you can probably think up a way to apply at least some of this approach to something else.

If you’re lucky some great song might even inspire you to excel with Monarch.

Finding the Needles in the Haystack

One of the main reasons that we use data extraction tools is to help us make decisions based on that data. And while the entire data set is certainly useful, often it’s the outliers, the values that are distant from the rest, or the values that exceed certain upper or lower known thresholds, that interest us the most.

Monarch helps us find those items with its filters feature. When in the Table window, we can ask Monarch to show us only those records that match what we’re looking for.

Here are some typical cases where filters excel at mining data:

1. Isolating regional data: “Show me all of the sales for the Western region only, not everything nationally.”
2. Highlighting good or poor results: “Show me all of the products on which we’re earning greater than 40% margins.”
3. Spotlighting individuals: “Show me Bob’s sales.”
4. Limiting the data to specific time periods: “Show me the second quarter AR collections from within this annual report.”

You create filters by creating Excel-like formulas that Monarch calls expressions. These expressions are often comprised of field names, logical operators LINK (>, <, =, etc.), and specific values.

You can also include Monarch’s functions (either the built-in functions or your own custom functions) in your expression.

Making the Rules

Monarch requires that you name the filters you define. Take advantage of this by naming the filter to clearly reflect what it represents. Avoid names like “AR Filter”. Instead, use names like “Southern region sales” or “Great Performers” or “Credit Notes Issued”.

Once you have created at least two filters you have the option of creating what Monarch calls a compound filter. That is, you can combine the filters to see the data that those individual filtering rules create.

Monarch allows you to use either the OR Boolean operator or the AND Boolean operator. When you specify that any of the filters you’ve listed satisfy the filter, Monarch imposes the OR operator. When you specify that all of the filters you’ve listed satisfy the filter, Monarch applies the AND operator.

It’s important to note that you cannot combine the OR and AND operators when defining a compound filter.

For instance, a compound filter defined as “Southern region sales” AND “Great Performers” would show you the great performers in the Southern region. A filter named “Active region sales” defined as “Southern region sales” OR “Northern region sales” OR “Western region sales” would include everything but the Eastern sales, if the sales were all defined in that manner.

Breaking the Rules

Here’s where things get interesting. You can define a compound filter made up of a combination of a single, or a number of, compound filters and a single, or a number of, formula-based filters.

Now you can define “Great Active Region Sales” as a compound filter using the “Active region sales” (which uses the OR operator) compound filter and the “Great Performers” formula-based filter, tied together with an AND operator.

This is an example of how you can effectively combine the OR and AND operators in one filter.

Of course complex filters may take a longer time to process than simple filters during data extraction, depending on the volume of data in question.

Filters enable you to mine that golden information in a pile of raw data and let you really excel with Monarch.