As we’re at the midpoint of the 30 Days to Become a Better Monarch Modeler series, today on day 15 I have a real-life challenge and proposed solution that I think quite nicely reinforces some of the concepts and tools that we’ve discussed so far.

This week a new Monarch Forum member named MikeMetta posted a sample multi-column address list and asked for the forum’s assistance in creating a model to handle the extraction properly, as he wasn’t yet achieving the results that he needed. It was, in Mike’s own words, “tricky”.

Looking at his sample it was clear that Monarch, using its regular address handling facilities would have a tough time with it, due to the now-you-see-it-now-you-don’t nature of some descriptive comments that appeared within the addresses. If that weren’t enough, telephone numbers and web site addresses were tacked onto the address without any blank lines in between that information and the address.

What to do?

Instead of trying to extract different pieces with detail and append templates as was being discussed as possible approaches, I began working on capturing the entire block as a single field, with the intention of breaking it up by using a few different calculations myself, and by letting Monarch do some of the heavy lifting too.

The method successfully extracted the sample data properly, so I thought that it might be interesting to share this with you, as this type of challenge, or variations of it, isn’t as unusual and in fact is probably pretty common.

Setting it up

First up was defining the multi-column region which was pretty straight-forward work for a two column layout. On to the detail template…

Looking at the sample, the largest block was six lines long. I created a detail template that was six lines long to handle this painting the single field across the width of the column, trapping on a single alpha character at the start of the field (a non-blank trap would work too, and would probably be better for numbered companies). In the Advanced tab of the field definition, I set the End Field On the end of left justification. To keep it easy for future formula building, I named the field simply as “A”, and that’s the end of the template building. On to the slice and dice work…

Breaking it down

The following employs functions that have already been cited in this series of posts.

In the Table window, deriving the Company name needed a Character field with the formula:

Now the real data separations start. In order to isolate the true address data, I thought it necessary to separate each component of the block, so line 2 of the block became the field named L2 with the formula:

Similarly, L3 and L4 with built with TextLine(A,3)and TextLine(A,4).

Now we’ve got the name and address bits broken out, but how do we get Monarch to know which lines are truly address related, and which are descriptive text? The unfortunate answer is that we can’t. At least, I can’t determine a universal solution to that. Still, where there’s a will there’s a way.

I added a user-edited calculated field to the model named Skip. Skip represents the number of lines to ignore in the address block after the company name, since all of the sample address showed descriptive lines, when they existed, between the company name and the address.

Then I keyed into the model a value of zero if no descriptive text existed for that record, or a value of one or two as applicable when descriptive text did exist, as appropriate.

With this prep work done, I could finally build a field that I could use to feed a Monarch Address Block. My AddBlock character field got this formula:

Using AddBlock as the data source for an Address Block, I instructed Monarch to extract Address line 2 as “Address”, City as City, Region as State, and Postal Code as Zip Code (for US addresses in the sample). There’s the heavy lifting done as Monarch automatically populated those tough to extract bits into the table.

Finally, the built the Telephone field with:

and the URL field with:

Your Task for Today

With a reasonably quick effort, I’d developed a single reusable model that was able to extract all of the important data. That said, can you see a shortfall in this model as has been described? What component isn’t clearly (and entirely) isolated?

Your task today may be tougher than the previous tasks, but you’re now equipped to complete it. Add the functionality to the model that I didn’t describe: duplicate this model and to it add the full descriptive text that is integrated in the original address chunk, if any.

Mission Accomplished

While I certainly wouldn’t expect that you’d want to use an approach that employed user-edited fields when hundreds of addresses are to be extracted, but for reasonably small samples it could be a workable long-term solution. It really shouldn’t take too long to populate the Skip field for even a fairly long list of addresses, and given the end result, that time spent doing manual data entry might be viewed as a good investment.

Admittedly we’ve had some bumps in the road, but I hope that you stay tuned for the second half of the 30 Days to Become a Better Monarch modeler, and discover even more that you can do to excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 16 of the series, or review Part 14.

I’ll be among the first to say that being a desk jockey can be tough on a person. Long days virtually tied to a desk really aren’t healthy, so after a bit of an extended absence from sports, I’ve recently returned to cycling to get more exercise. Being somewhat apprehensive, I was careful getting on my bike. But it’s true that you really don’t forget how to ride. Your confidence grows quickly and you develop a “can-do” attitude.

For day 5 of the 30 Days to Become a Better Monarch Modeler series we’ll get comfortable with the methods of defining and controlling field content for extracted fields, so as to be able to build our Monarch models to mine data from reports and other sources as quickly and as accurately as we can. Extraction fields differ significantly from the calculated fields with which we add new data to the data from the original data source.

Fields of Dreams

Extraction fields are defined while building templates, and the mechanism for controlling the action of the field are handled within a two tabbed dialog box. The first tab is for the General options, which includes:

• The name of the field, which can up to 62 characters in length. The supplied field name can control most characters including spaces and underscores, and both upper and lower case letters can be used. See the online help for the list of the few characters which are not permitted for field names.
• The type of content within the field. This can be:
  • Character,
  • Date/Time,
  • Numeric, or
  • Memo
• If the field is numeric, the number of decimal places can be specified (from 0 to 9 places are permitted).
• The format with which to display the data. There are several options, each of which is dependant upon the field type. Monarch guesses as to default to the proper data format based on the extracted content of the field, but this can be overridden.
  • For Character fields, only the option (which doesn’t make it an option then, does it?) available is General. Basically what you see is what you get with General formatting.
  • For Date/Time content, we have the following available formats (for which Monarch displays a short example for reference):
    • General,
    • Short date,
    • Long date,
    • Short date & time,
    • Long date & time, and
    • Time
  • For Numeric content, we have:
    • General, once again,
    • Thousands,
    • Currency,
    • Percentage, and
    • Time span
• The number of characters to display for this field, or its width.
• A checkbox to use if you want to hide the field in the Table window.
• The alignment style: left, center, or right alignment.
• The template width. Unlike the display width, this positive integer value indicates the number of characters to extract from the report. This option can be a little confusing, as the Monarch technical specifications indicate that a Memo field can contain over 65,000 characters. So you’d think that you can set number quite high for wide reports, such as when the report has over 1,000 columns for example. But you can’t. The upper limit for template width is only 254 characters.
• Then there are options on how to control the verification process, which helps to determine if the fields have been painted properly. You can elect to check the left and right sides of the field, as well as the field type.
• Finally, you can elect to leave empty cells blank, or to use the value from the field in the previous record.

I do have one additional tip related to the simple act of naming your fields that can save a lot of time and effort later on if you plan on using your extracted data in a database that can be programmed with SQL (such as Microsoft Access), and it’s this: don’t use reserved SQL keywords as field names.

That’s a bit easier said than done as there are a good number of SQL keywords.

It might not be your time and effort, but some programmer one day will thank you.

That brings up yet another tip: don’t use built-in Monarch function names as field names.

The confusion that you’ll experience when building and reviewing calculated fields and filters is completely avoidable. Instead of naming the dates from your Accounts Receivable report “Date”, name them “InvoiceDate” or “InvDate”. Anything but “Date”.

Keep in mind though that Monarch doesn’t care what you name your fields, so long as you’ve used legal characters to do so.

Advanced Monarch

On the Advanced tab of the dialog, there are two sections that affect the data extraction. The first section affects the control of where the field begins and contains the following options:

• A specific line number. This is based on the line on which the field appears within the template sample.
• A string, or a series of characters, that appears at any position within the line immediately prior to the line in which the field exists.
• A string that appears within the same line as the field and before the field.
• After the last defined field in the template, or more clearly, use this option for a field that appears two lines after the last line of multi-line field which is also extracted in this template.

The second section controls how to terminate the field, including:

• After a specific number of lines.
• Following a non-blank series of n characters which end immediately prior to the beginning of the field, in a row below the start of the field.
• Following n blank field values. This refers to the position on the line that has been painted for the field. For instance, if you specify 2 blank field values, then a section of text that has a single blank line in the middle will be captured in its entirety, until such time as there are 2 blank portions for that position in the line.
• Until the end of left justification is reached. This is often the method to choose when defining text to use in conjunction with the address block feature. Just be careful that the last line doesn’t have a leading space, as Monarch will miss that line in that case.
• The classic “none of the above”, which really means that Monarch should capture this field until it runs into data which meets the trapping criteria for another template. It’s a bit of an odd choice, as Monarch will always enforce this rule regardless of the “end field on” option that was selected.

Your Task for Today

I’m almost 100% sure that when you first began to learn and use Monarch you used the sample reports that are provided in conjunction with the supplied Learning Guide. But after going through it once (maybe some time ago?) you checked that task off on your to-do list and moved on.

Today I want you to open each of the sample reports and build a model for each report to do basic extractions. Do this without using the Learning Guide. Base your activities only on what we’ve covered to date in the 30 Days to Become a Better Monarch Modeler series.

If you run into content that you can’t model without venturing into Monarch’s other abilities, skip it for now.

Define Your Success

Life is like riding a bicycle – in order to keep your balance, you must keep moving. ~Albert Einstein

Building Monarch models gets a bit tricky at times because there are so many variations and possibilities in the reports that each of us use regularly. You simply might not run into the need to use all of the Advance field tools in your frequent model building tasks.

But by practicing a little, and being aware of the results that each option can provide you, you’ll be able to build models accurately and quickly. Keep moving forward and you’ll soon excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 6 of the series, or review Part 4.

Many of the reports that our various computer systems generate for us present data in a very linear fashion. These reports tend to be structured in a very top-down manner, having a layout similar to:

• Header
• Optional grouping data (what we as Monarch modelers normally think of as append data)
• A cluster of detail records
• Optional subtotals
• Optional new groupings, followed by more details and subtotals
• Finally, there might be footer data

In fact, that’s probably the most common presentation.

But what about the rest of the reports that we use? What layout do they tend to have?

Typically those reports either scatter data so that the whole page, or most of the page, is a single detail record, as is often seen with special purpose forms, or they display the data in a newspaper-like flowing column format. The latter is what Monarch calls “Multi-Column Regions”, or MCR, and that’s the focus of day 4 of the 30 Days to Become a Better Monarch Modeler series.

Break Down the Boundaries

There are a few steps to setting Monarch up for MCR work.

1. Review the document to see if there’s anything unique about the start of the MCR area within the report.
2. In the Report window, under the Template menu, select Multi-Colum Region…
3. Activate the MCR tools by clicking the MCR “is active” checkbox.
4. Tell Monarch how many column regions exist within the report as you move horizontally across the page. You must define at least two columns, even though Monarch allows a value of 1 for the number of column. Also, you cannot define more than 40 columns.
5. Determine the starting position of the leftmost column
6. Set the width of each column.

You want the columns to be wide enough to capture every field within the group. Assuming that there are three fields within each of four columns, you want the column width to be sufficient to paint the three fields, not all twelve within the line (as you would with a conventional detail template).

Notice that the dialog box also has options for the “Boundaries (vertical)”. These are how we instruct Monarch to limit the MCR area to only specific zones within the report.

We can use the defaults of the top of the page and the bottom of the page, but we can also use previously defined templates as our start and end points. Unless the entire report is an MCR area then we should, at a minimum, set the start of the MCR area with a template.

When we close the MCR dialog by clicking the OK button, we now see that Monarch displays a type of a red frame around each column, and it will treat the data contained in these regions differently that it would without the MCR feature activated.

Now we’ll handle the task of building a template to extract detail records from within the MCR area.

Select a detail line from the report as you would normally, by clicking just to the left of the line(s) that you want to use for your template, and create a new template.

What’s wrong with the sample line? Monarch didn’t bring the content of the whole row in as it does normally? Fear not, there’s nothing wrong. It’s only showing the fields available in a single (the first) column.

Now trap and paint fields as you would normally, but when the new template definition is saved, it’s applied to all of the columns in the MCR area.

New to Monarch V10 is the Column() function which, when used with MCR-derived data, returns the MCR column number in which the data appears on the page.

Your Task for Today

If you’ve not had to model a report with wrapping columns previously, today’s the day that you’ll gain some experience with this aspect of report mining. If you don’t happen to have a report that has this style handy, don’t worry – you can quickly whip up a basic layout in Excel.

First, create a blank Excel workbook and change the font for the entire sheet to Courier New. Now populate columns A through F with some sample (randomly made up) data in, say, the first 10 rows only. Keep in mind that when designing reports most programmers will ensure that the columns maintain a repeating fixed width for the columns, so that each column is maybe 20 characters wide, for instance. Do what a report programmer would do and be consistent.

Now you can save this sheet as a “Formatted Text (space delimited)” prn file type with Excel, which you’ll be able to open in Monarch to experiment with. First define the MCR template, and then set out to define a detail template.

When you’ve successfully captured your sample data in the Table window and add a calculated, formula-based Numeric field employing the Column() function. When that’s done, go back to the Report window and experiment with changing the parameters in the MCR definition to see what happens. Be sure to revisit the Table window to see what effect your changes have made. Do this even if you did have a good MCR candidate report file handy and didn’t resort to creating your own with Excel.

Subscribe and be Involved

We’re approaching the next stage of the 30 Days to Become a Better Monarch Modeler series as we’ll move on from some of the more fundamental aspects of Monarch. And while some of what has been covered so far is old hat to some, every day many more people are just being introduced to Monarch, so I felt it important to include these topics in the series.

Make sure that you don’t miss any of the remainder of the series; use the “Keep up to date” section above to subscribe to the free service that emails new ExcelWithMonarch.com site updates directly to you when they’re posted. I invite you to take a moment and post a comment below to share your thoughts on this post, the series, or the site in general. Or if you’d prefer, email me directly.

Is your data on a long and winding road?

Just because your data is twisting and turning all over the place doesn’t mean that you can’t control it. Just use the Multi-Column Region template building feature, and you’ll be headed in the right direction as you excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 5 of the series, or review Part 3.

And once again, after an unfortunate and unexpected interruption caused by my site host, ExcelWithMonarch.com returns. It was a bit like a radio station whose transmission tower is destroyed in a storm or something.

Though the outage was out of my control, just days ago I’d promised daily posts for this series. Timing is everything, right? I swear, if it weren’t my bad luck I’ve have no luck at all. Nonetheless, I’m pleased that you’ve returned, so let’s carry on then, shall we? Onwards and upwards…

There was a moment in a recent television drama episode wherein the hero was struggling to find clues to solve a mystery, and it occurred to him that two seemingly unrelated documents were in fact related. He stacked the two sheets as if he were to staple them together. He noted that one document that had some elliptical graphic elements above a text message, and proceeded to cut holes where the graphics were and the holes then revealed a meaningful message in the text message on the second page.

That’s what Monarch does with your report when you define templates: it highlights the data that you want to see, and effectively ignores, or obscures, the rest.

In order to for a model to extract data from a report, fields must be defined in a template.

Monarch offers four types of templates in its toolkit, and on day 3 of the 30 Days to Become a Better Monarch Modeler series, we’ll examine how each of the templates work together, and when to employ with template type to accomplish specific tasks.

The Detail Template

The detail template is the core of Monarch’s report mining abilities, and every model that has a report file as its source must have one (and only one) detail template in order to extract data from the report.

Detail templates are generally used for the fundamental basis of the report; its raison d’être. Whether it’s an inventory list, a sales report, an accounts payable report, or a financial statement, reports tend to have a central topic, and detail templates intend to hunt for the information that relates most closely to that topic.

The Page Header Template

When you want to capture data that appear at the top of the report pages in addition to other places in the layout of the report, you want to use a page header template. Like the detail template, only one page header template can be defined per model.

Page header templates have another role though. You may find that on some reports detail records carry over from one page to another with a complete page heading such as the report title or run data and column heading appearing right in the middle of the record.

In such cases, Monarch won’t “see” the last part of the detail record and won’t capture it properly. But there’s a solution to this problem.

To ensure that records that are interrupted by a page header are properly captured, it is necessary to define a proper page header template even when you don’t intend to extract any data from within this template.

Once the page header template is defined, Monarch will capture the entire record properly.

The Append Template

A Monarch model can contain multiple Append templates. Append templates are intended to add data to the data captured by Detail template which physically appear on the report prior to the detail data. I say “intended” because there is at least one invaluable technique, coined as the “guru trap”, which is used to append data that appears subsequent to one detail record, and prior to another.

The Footer Template

Data that is extracted with a footer template is added to the detail data. Footer data will be added to all detail data that was extracted since the last instance of qualifying footer data.

Within the report, footer data appears subsequent to detail records, and perhaps prior to append or additional detail data.

Unlike the detail and page header templates, multiple append and footer templates can be created within a single model, but to a maximum of 20 total templates of these types.

Your task for today

Becoming a better Monarch modeler requires familiarity with each of the available four template types. Today you need to build a new model which utilizes each of the template styles.

First, examine the report and determine which aspects of the report should be handled by which type of template. Now build each type of template. My experience is that not every report has data that can or should be handled with a footer template, so unless you have a good candidate for this modeling at the ready, you may want to resort to one of the sample reports that are installed alongside Monarch for this exercise.

Your template for success

While you might not be attempting to solve mysteries and perform espionage with your models, a solid grasp of the available templates will help you excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 4 of the series, or review Part 2.

Put some nice smelly cheese into a trap, and you might catch a mouse. Put the right trap into a Monarch template and you’ll capture your data.

On day 2 of the 30 Days to Become a Better Monarch Modeler series we’re going to review the tools that are Monarch makes available to us to let us teach it where our data is hiding.

Many software tools are available for handling databases, and Monarch Pro can perform that work as well as or better than many others, but its real specialty is report mining: extracting data from report files and in the process converting them from static documents into dynamic tools for business intelligence.

Central to being able to extract report data is the ability to differentiate the data to be extracted from the rest of the report. Monarch refers to this as trapping the data. You use special controls or characters to direct Monarch to isolate key fields from the remainder of the report.

	Replaces the text in the sample line with the currently selected rows in the report window. This is often useful when multiple fields are required within a row, but not every field is populated within each row.
	New to Monarch v10 is the ability for Monarch to generate the traps for you. This is activated by selecting the Auto Define Trap wizard.
	Use the Alpha trap to tell Monarch that you want to include rows which have an alpha character (a-z) in the selected position.
	Use the Numeric trap to tell Monarch that you want to include rows which have a numeric character (0-9) in the selected position.
	Use the Blank trap to tell Monarch that you want to include rows which do not have any characters in the selected position.
	Use the Non-blank trap to tell Monarch that you want to include rows which have any character (a-z, 0-9, or any other symbols) in the selected position, so long as the selected position is not blank.
	Often used in multiple positions the Numeric OR tells Monarch that you want to include rows that have numeric values (0-9) in either this position OR that position OR that position OR …
	The Postal trap inserts <ZIP> into position 1. When you use the Postal trap, all of the fields that you define must align vertically; they must begin and end on the same positions within the row.
	The NOT toggle will reverse the functionality of the selected trap character(s). The trap characters will appear in red with a strikeout font. Effectively, this tells Monarch to include rows that do not have the selected trap in this position.
	Clicking the Reset Trap icon will remove any traps that have been added to the trap line.
	The Auto-Define Fields icon will paint all of the fields that Monarch detects in the sample area. The data types will be automatically assigned based on the content of the field. Monarch does a very good, if not always perfect, job of detecting fields.
	When the cursor is in the painted area of a field in the sample area, the Field Properties icon will be available to you. Of course, you could just right-click or double click the field to bring the up field properties dialog.
	When the cursor is in the painted area of a field in the sample area, the Reset Selected Field icon will erase the selected field.
	The Reset All Fields icon will erase all of the painted fields in the sample area. After your confirmation, of course.
	Clicking the Pull Left and Push Right icons will move either traps or fields over one position, depending upon whether the cursor is in the trap line or the sample line. A nice enhancement to v10 is that the shifting acts slightly differently depending on the cursor location. If it’s within a field, only the selected field will move. If however the cursor is between fields, all of the painted fields will shift. This enhancement makes editing templates a snap.

Additionally, you can supply literals as trap characters. So if you know that you need to include all rows that have the string “CUSTOMER:” starting in position 1, then type CUSTOMER: into the trap line.

By the way, Monarch won’t interpret the word CUSTOMER: as a word. Instead, it looks at the characters individually: a C in position 1, a U in position 2, an S in position 3, and so on, up to the colon in position 9.

Use as many trap characters as necessary to have Monarch properly isolate your data from the remainder of the report. Your trap line can include a combination of characters inserted by using the buttons and characters that you supply manually.

Additional trap options

Under the Options menu you’ll find Input options for further control of the trap facilities. You can choose to make your trap characters case sensitive so that a trap that reads “CUSTOMER” will ignore occurrences of “Customer”. You can also choose to use a different set of characters for the traps generated by the buttons, such as Numeric or Alpha traps, by selecting the “Alternate” trap characters instead of the “Standard” ones. People often use the alternate characters when they intentionally want to trap on characters that exist in the report that Monarch normally uses for trapping purposes, such as the vertical bar, or pipe character “|” which Monarch uses for detecting numeric OR conditions.

Monarch v10 users can take advantage of a feature not available in earlier versions. New to v10 is Monarch’s ability to build the trap line for you. Hardened Monarch veterans might say that this is akin to removing the art from the science.

But just as we can drive our vehicles from A to B, it doesn’t mean that we all understand what’s going on under the hood. But sometimes the vehicle breaks down, so it helps to have a bit of knowledge about how it works and where things are. So you v10 users (especially the new-to-Monarch-with-v10-folks), don’t get too reliant upon the tool. Learn how to trap on your own and you’ll thank yourself one day.

Your task for today

Get comfortable with Monarch trapping tools. Spend at least 30 minutes with a somewhat complex report experimenting with different ways to capture different data sets within the report. Start by capturing as many similar records as you can – all of the rows that you would consider details – and then narrowing the data selected not by using filters but by only changing the traps. Practice until you can correctly select the proper trap tools as easily as you can type your name.

Trapping is critical

Many years ago when the exploration of the world was just beginning, trappers were not just looking to survive, but were often key players for international trade. Being a good trapper today can be of equal importance and is absolutely necessary if you want to excel with Monarch.

—

Continue your commitment to Become a Better Monarch Modeler with Part 3 of the series, or review Part 1.

In the last post we talked about the process of learning the fundamentals of Monarch, and this leads nicely into a discussion of Monarch’s unwritten (?) cardinal rule and how we can take advantage of it.

What’s the rule? Simply put, “Don’t allow the user to alter the original data in any way with Monarch”.

What’s the effect of the rule? No risk.

There’s no way that you can corrupt your report file with Monarch when learning how to model reports. That’s entirely unlike the messing around that I did with early spreadsheets and databases. You’ll never lose data or edit the original values permanently. Modeling is totally safe.

Just be carefully when exporting.

And the knowledge that you can do no wrong affords you great comfort.

You can try new things. You can experiment. You can fail. And if you do, all is not lost, save for perhaps a few minutes of your time.

But you may very well succeed, and when you do it’ll make all of your efforts worthwhile.

The Big Win

Such was the case for Grant Perkins, the UK-based consultant who pioneered an exceedingly useful little trick that’s now referred to as the “guru trap”. It’s a technique that permits a model to use append templates to add data to a record when the appended data physically appears subsequent to – not prior to, as is normally the case with append templates – the detail record displayed in the report.

Grant has contributed to ExcelWithMonarch.com in the past, and I’m pleased to have his input for you again today. Take it away, Grant…

I’d heard about Monarch and picked up a copy and spent a year or so using it on and off on a few projects. When a new reporting challenge was presented I looked at Monarch first to see if I could get the data from standard reports (lots of complex time calculations already done there) rather than reinvent the wheel. Often I could, sometimes I couldn’t.

A couple of years on, I extended the use somewhat when we needed to retrieve data, including a lot of long form text, from an external supplier’s system. The only method available to recover the information was to screen scrape using a screen emulator on the dumb terminal/dial up connection they had provided. A colleague who had authorized access to the supplier system wrote the screen scraping script and we ended up with a large file of screen dumps, mostly with data consistently positioned but there were a few exceptions that required conditional field calculations to extract and format ready for upload to our database.

Hmm. Monarch could be deployed much more widely than I had imagined.

Soon after that I discovered the Monarch forum and shortly after that was pleased to be provided with a copy of the CD that was made available to attendees of the 2003 Datawatch User Conference. On that CD were some presentations called ‘Voodoo Monarch’ that looked rather interesting …

Suddenly it was as if a door had opened and the lights came up bright. There was SO MUCH more one could do with the features and functions in Monarch than I had previously identified.

So I started to experiment hoping for success but knowing that failure would be a valuable learning experience anyway and cost little time.

The approach I adopted was to look at a problem report situation and come up with a few ideas that might or might not work. Off-the-wall was good for seeking out these ‘Voodoo’ approaches. Sometimes I would find something that worked without being sure why it worked.

As the ideas are applied to more and more examples of problems (forums are a great source of learning material) one can start to work out what is going on and perhaps even why!

The Guru trap, as it became known, had its origins in a rather nasty little data dump file (rather than a report whose appearance, nasty or otherwise, had been “designed in”) in which the master detail record was, at a minimum, about 4 lines long but some records could be several lines longer. Add to that that fields were not always present in all records or, apparently, in the same order and it looked like a completely unintelligible mess.

But two Monarch features came together to make the extraction viable in a single pass. A single pass through the report is always a good objective though there is no reason at all to stick inflexibly to that ideal if multiple passes through the report simplify the model development task and produce a quicker and/or more flexible result.

Firstly the undocumented one:

There is nothing in the processing of a report to prevent two or more templates sharing the same trap or indeed extracting the same ‘data’.

By sharing the same trap as a Detail template an Append template is, effectively, forced to reset every time Monarch finds a new detail record. In effect, this allows detail records to be extended beyond the notional limit of the number of lines that can be included in the trap sample. (This is counter intuitive to the ‘rule’ that Append data must always appear BEFORE detail data, although in fact rules in the design philosophy are not broken – just adapted slightly to suit the purpose. Well, after all, what are rules for?)

That means that if your shortest record has 3 data lines and a blank line before the next record, the sample ‘mask’ cannot be longer than 4 lines. Records with 12 lines therefore become a problem. Being able to create a ‘stack’ of template masks based on the same starting point selection became the objective from an idea that appeared from nowhere one day and proved to work rather well. (It may have been accidental at the time, but who cares if something works?)

The documented feature was the use of the ‘Start Field On’ Advanced Field Property and its ‘Preceding String’ option.

This meant that so long as the content of the report had enough information to uniquely identify the field required, the data could exist anywhere between the current template trap and the next occurrence of the trap. Specifically, as used for this concept, that meant anywhere between the start of one detail record and the start of the next.

It was actually several months before I understood well enough why things worked to be able to start to explain them if anyone asked (and to be confident that they would work reliably!)

Over time a few more possibilities turned up and enhancements in later versions of Monarch added to the array of features and functions available but the primary principle of discovery remains: Try it and see what happens and what new ideas the attempt presents you with.

Visualize the report and gain an understanding of the problem in terms of what you would need to do to get each field, or group of fields, how you need them to be. If Monarch does not immediately present an obvious solution ask yourself what the solution would be and then consider whether there are function(s) or features that might possible be helpful to move you closer to what you need.

[I find that reading the Function(s) help is very useful for keeping possibilities current in my memory as well as for prompting new ideas to try out. Often they are not practicable but sometimes they lead on to success.]

If you get to a “If I could do this ….” situation it then becomes much easier to work out how, using Monarch, you might get there. Trying it out usually costs little time and teaches a lot.

Be prepared to go back a step or two in your efforts.

Finally, be prepared to try something completely different. On any journey looking for a way around a wall is often more successful than attempting to knock it down.

Thanks Grant!

Defining a “guru trap” requires that you employ exactly the same trap characters in the append template as were used in the detail template, in exactly the same positions. This is done regardless of the representative data within the append template sample line.

Next, the field(s) to be captured is/are painted in the sample line. In the advanced options for each field, either of the “String _________ anywhere in the previous line” or the “Preceding string: _________ in current line” options is employed as is necessary, based on the report content.

There’s usually some string literal, like “NOTE:” or perhaps a / character that appears in a date in one of those lines that can be used in the advanced options of the field definition.

This technique is most often used when the data that needs to be appended appears only intermittently – perhaps as infrequently as only once – amongst an otherwise contiguous series of detail rows.

Grant’s experimentation efforts in developing this technique have enabled many reasonably straightforward (now) modeling solutions, which may well have been impossible if not exceedingly challenging without it.

Your experimentation may not be quite so radical. Maybe it will be more along the lines of trying out new functions, or combinations of functions, that you’ve not previously attempted.

Why so serious?

So go ahead and experiment. Have some fun. Risk little. And, above all, continue to find new ways in which you can excel with Monarch.

From time to time you may run into a report designed by some renegade programmer who thought that having values shift positions from line to line on the report was actually a good idea. Additionally, maybe she thought that, rather than display zero values, it would be best if the report simply doesn’t display anything at all for that field.When it comes to making it easy for the Monarch modeler, nothing could be further from the truth. We like predictable (in that we know that a value – something! – will appear where it should) and nicely aligned values.

When the Floating Trap Isn’t Enough

Sometimes you can handle these situations nicely with the floating trap feature. But sometimes there just too many problems with the line for Monarch to make sense out of it.

Fear not! All is not lost. But it will take a little extra work to get the results you need.

Recently a user of Datawatch’s Monarch Forum posted this snippet from his report:

Hmm. This sample doesn’t post very well here. Rather than spending my time fighting with it, I’ll point out the original post with the sample.

You can see that there are both floating and missing values in the sample. The best of both worlds, as it were…

The solution involves knowing how many fields you expect to see on the line, and introducing a device – an unusual character, anything that you wouldn’t expect to normally see in the report – to allow for extracting the values easily with one of Monarch useful functions. This character will allow us to split up the fields later on.

The first step is to capture the entire line as a single field, rather than even trying to get this field and that field at the outset. We’ll name that field “A” as it really doesn’t matter what we call it.

Next, we insert our splitting character. Field B has the expression:

In field A, any occurance of 11 consecutive spaces will be replaced with a single vertical bar, otherwise known as a pipe character. Why 11 spaces? Any time there are 11 spaces in this sample, we know that a needed field is completely missing. You’ll need to carefully assess your particular report to determine this number.

Next, we clean things up a little bit more. We’ll still have a number of repeated spaces, so determining exactly where our fields are is still a bit of a challenge. Now we eliminate the repeated space, replacing them with just a single space. Field C has the simple expression:

Now it’s a simple matter to replace all of those unwanted spaces with the splitting device (our delimiter). Field D has the expression:

Be a Data Chef: Slice and Dice!

Finally we have string with nothing but values and pipe characters. Monarch can dissect values from such a string easily. Our first field that we really want from the report, Open, is a Numeric field and has the expression:

Monarch will split field D into 40 sections, based on the location of the pipe characters. You can always supply a higher number of sections than you really need. Monarch doesn’t care. Just make sure that you’ve supplied enough. The exact number is fine if you want to count the fields.

Once Monarch extracts that bit of the original field, it’s still a character string, so we use the Val() function to convert it to a number, as that’s expected by the field type we defined.

The next field, NetBilled, simply increases the last value in the last formula by one, like this:

By spending a few minutes creating the remaining fields (use the Duplicate button), you’ll have tamed this beast of a report that you thought was not possible to model with Monarch.

The process could be streamlined a fair bit by combining the separate calculated fields. Instead of using the various fields B, C, and D as presented above, you could establish field B as:

Then your final fields would use:

Additionally, by introducing just a couple more fields, you could make this a portable solution for any report that is giving you the same problem, and the only work you need to do to model for that particular report would be to define the final values fields themselves.

By using runtime parameter calculated fields for the number of spaces (SpaceCount), the delimiter character (NewDelimiter), and maybe the number of fields if you really want to be picky, the final expression would be:

Alternatively, Let Excel Split The Fields

Building the various numeric fields that split our B field (with all of the pipe characters), might be avoidable. And you can achieve with the same end result too!

Just export the B field, and any other fields you’ve appended, to an Excel file or text file, then import that file into Excel and use Excel’s Text to Columns feature, specifying the pipe character as the Other delimiter. Excel will put each field value in its own cell properly across the columns.

Now that’s using Excel with Monarch!

Powerful Data Mining Technology for Everyone

It’s easy to take your tools for granted, especially after you’ve been using those tools for awhile. But every time I see a really oddly designed report, and someone has created a Monarch model for it, I’m reminded of just how good Monarch really is.

Combining many of Monarch’s features, and perhaps using them in unusual or even possibly an unintended manner, will really let you excel with Monarch.

Video: Monarch’s Multi-Column Region

Every business uses some form of accounting software. Of course, just as there are different sizes of firms, there are large scale and small scale accounting software solutions. Similarly, not every firm uses a complex ERP solution; more often than not they’re using software purchased at a local office supply or electronics shop.

According to Alan Salmon & Associates, who write on the topic of accounting technology, in Canada alone there are literally hundreds of thousands of users of just two popular and inexpensive accounting packages: Sage’s Simply Accounting and Intuit’s Quickbooks.

As I see it, that’s hundreds of thousands of users that all share…

The Single Problem with Accounting Software

Both of these packages offer a wealth of features, not the least of which is the number of reports included to help with not only the pure accounting, but also with the general management of the business.

Over time, these packages can develop into pretty comprehensive databases. They usually track information on customers and vendors, employees, sales, expenses, inventory and more.

Users often become well acquainted with the available reports, and know what content is hiding where. The adventurous may attempt export data from the package to another format, such as Excel, with decidedly mixed results.

Frequently the growing business will find that its information needs are expanding beyond the abilities of the software in use. There’s nothing wrong with what’s being captured; the business just needs better access to the data for stronger and more flexible intelligence on the business.

That’s where accounting software packages usually develop the same problem: they don’t always give the business the management tools it needs, so you give up and try again with a replacement. More often than not, the replacement doesn’t get you any further ahead.

Thousands of Problems Need Only One Solution

Wouldn’t it be great if you could leverage your existing accounting solution to give you the full-blown information system you really need?

The answer is simple. Just add Monarch to the accounting system.

Just about every major current accounting software package lets you print its reports to electronic text files. Monarch can easily convert your static data in those files into a flexible, efficient, business management machine.

Free your business of your accounting package’s limitations. Let Monarch help you find key data, develop custom metrics, and summarize what you want how you want it. You can even link data from other sources outside of your accounting software to build your own management tools and add even more value to what you already own.

I’ve prepared a simple example of how you can convert an interesting, if difficult to use, report into actionable data. This video spotlights Monarch’s ability to easily extract data from data in multiple columns. Watch the video now.

You Can’t Always Get What You Want

Regardless of the size, or for that matter the cost, of your accounting program, you can make good use of Monarch to make the most of your existing software solution. No system is perfect; nothing will deliver everything that you’d really like it to do. But you can get what you want, and more importantly, what you need, when you choose to excel with Monarch.

How would you describe a painting by Da Vinci, Renoir or Monet? A picture truly is worth a thousand words.

Similarly, I could go on and on describing the benefits of Monarch to someone who has never seen it action, but it’s just not the same as seeing for yourself.

So, to that end, I’m pleased to present what I believe to be is the first freely available web video for building a Monarch model (6MB). It runs almost exactly four minutes long. There’s no audio. I need to shop for a decent mic. Any excuse to shop for gadgets!

The video shows how to extract data from a typical list style report, and how you can quickly add summary information that doesn’t appear in the report.

Producing videos such as this is a bit of undertaking, so before I go off the deep end preparing material in which I think you might be interested, it would be really helpful if you just told me. I’m open to suggestions!

Post a comment below, or drop me a note at sandy DOT cavalaris AT excelwithmonarch DOT com (sorry for the lack of a link – trying to avoid the spam bots as much as possible as they seem to have found the site already).

And if you’ve never seen Monarch software in action before this, what do you think? Did you immediately think of a few ways to put it to good use?

Personally, I think the idea of one day being able to work faster and smarter probably put the smile on Mona Lisa’s face. Are you ready to excel with Monarch?

Mining for Gold

In all likelihood, Monarch is most often used to extract data from report files. After all, that’s its specialty.

Unfortunately, this most fundamental aspect is often the program’s most challenging task. Sometimes it seems that it’s more art than science. Many would say that labeling this process as a “craft” would be more accurate, and I’m inclined to agree.

Just as a fine craftsman, say a wood carver, knows exactly which tool to use to execute his vision, you’ll need to be intimately acquainted with the template process and tools. Of course, you won’t need to use every tool every time, but you’ll know which tool you need to use and how to use it. Seem reasonable?

Let’s look at what’s available to us in v9 Pro. The template window has five areas:

1. The template name and buttons
2. The template type information
3. The tools area
4. The trap line
5. The sample line(s)

Area 1

Always take a moment to properly name your template. It might not seem that important for simple models, but you’ll be glad you did it right when your model grows, or especially when you’re using object linking later on. Be specific. What will “Detail” mean when you have 100 models that all have a “Detail” template? Wouldn’t “Shipping Report Details” be better?

If your template is even slightly complicated, take a moment and document it with a comment. Explain why it is setup the way it is. Even you might not understand it if you come back to it in six months.

We’ll look at links in detail in a later post.

Area 2

What’s the purpose of these different template types?

• Page Header: used to extract data which occurs at the top of every page, like page numbers or run dates. (Or to create a virtual page if the need arises.)
• Append: used to extract data within the body of the report that we want to associate to the detail data. Append data usually appears before the detail data on the report, though there’s a neat trick you can use to use an Append template to capture data which appears after your detail data.
• Detail: used to capture the lowest level of detail in your report. This is generally the basic information that you expect to appear for each record you want to extract.
• Footer: used to capture data that appears after your detail data.

The checkbox for Floating Trap lets you tell Monarch that the trap you’ve defined (we’ll talk about that next) might move around within the line on the report a bit, so don’t always expect it to be exactly in the column position where you defined it.

Area 3

This is where the bulk of the tools lie.

The text box for the Trap Line refers to the line number of your sample in Area 5. Typically report layouts mean you’ll be trapping based on the first line of the sample, so the default value of 1 will be correct. But sometimes the best way to trap will be based on another line in the sample, so be sure to change this value to be the correct line number, or your trap won’t work as you intended.

The next icon, Replace Sample Text, lets you change the sample appearing in Area 5. Perhaps you selected four lines, but need five. No problem. Simply select the five lines you from the report that you want in the sample, and then click the Replace Sample Text icon. Area 5 updates based on your selection. This is also a good tool if your report changes to add, move or remove fields. The most important thing about the sample you use is that it defines the minimum number of lines between occurrences of the template trap line in the report. By comparison the data in the sample is not at all important EXCEPT when working with floating traps (for trapping purposes) or if you want Monarch to autocreate fields for you. I’ll come back to these topics in a later post.

The Alpha trap puts an alpha character in the trap line on the current column position. Typically you’d first click in Area 4, the trap line, where you want the trap to appear, and then click the trap tool (Alpha, Numeric, Blank, Non-blank, Numeric OR, Postal Trap or Not Trap icon.

You use the Alpha trap to tell Monarch that you expect a letter to appear in this position.

The Numeric trap tells Monarch that you expect a number in this position.

The Blank trap tells Monarch that you only want lines where nothing appears in this position.

Alternatively, the Non-blank trap tells Monarch that you want lines where something, anything but blank, appears in this position.

You use the Numeric OR trap anytime you expect to see at least one digit in any number of specific positions. As long a number appears in any one of those positions, Monarch will trap that line.

The Postal trap inserts “<ZIP>” into the trap line at the leftmost position. This trap icon is really only still around to support older models, and was designed to help extract address data. This trap functionality has been replaced by the “Address Block” feature, which you should use instead, so I won’t elaborate on the Postal trap further. We’ll cover the Address Block feature later on, I’m sure.

To use the Not trap, you first highlight the trap in the trap line (Area 4). When you do that the Not icon will change from disabled (nothing happens when you click it) to enabled. Clicking the Not trap will change your trap from black font to red font, indicating that Monarch will only trap when the line, in the indicated position, does not show what the trap is. You need to be a bit careful when using the Not trap in multiple positions, or you may not get the results you expect.

The next icon Reset Trap will clear the trap line in a single click. Start over!

The next set of icons relate to the sample area (Area 5). The Auto-Define Fields icon will take an educated guess at which fields in your sample area you want to capture based on your trap. While often useful, the results aren’t always perfect. It does work quite well, though, as long as your fields are preceded with labels ending in a colon character, like “Name:”.

The next icon displays the field properties of the active field. You must either paint (we’ll talk about that shortly) a new field, or click in an existing field to use this icon.

The next icons erase defined field areas. You use these if you don’t want to capture the defined field(s).

Finally the arrow icons allow you pull your defined traps and fields to the left one position, or push them to the right one position. When you pull left, anything right of the current cursor position shifts to the left, anything left of the current position doesn’t move. When you push right, anything right of the current cursor position shifts right one position and anything left of the current position doesn’t move.

Area 4

Now that we’ve reviewed how the various tool icons insert characters into the trap line, you need to know that you’re not limited to those traps. In fact, you often don’t need to use any of them to trap properly.

Sometimes you can just type words or symbols (literals) in the right position. You might have a trap as “Customer”. Or a date trap that uses forward slashes only to trap dates like 02/01/08, knowing that two / characters separated by two spaces will do the trick.

Some tricky reports require just the right combination of trap tools and literals to get what you want and only what you want. This is where the craft comes in (you knew that had to come up again, didn’t you?).

Trapping well requires that you recognize the specific recurring patterns within your report that allow you to differentiate a data line from another irrelevant line on the report.

Experienced Monarch users will probably chuckle a little at this next comment. When you start modeling a number of reports, you may well find that when someone hands you a printed report you don’t initially read it for its content, you’re scanning to see what you can trap. Or maybe that’s just me. LOL!

Sometimes you may find that because of the design of the report it is impossible to build traps that will capture what you want without picking up some that data that you don’t want. Or it’s simply taking too long to model that particularity. No problem. Go ahead and capture everything. Monarch provides other tools to solve that problem, but that’s a discussion for another day.

Area 5

You’ve worked hard to isolate the data within the report. Now’s the time to define which data you want to capture within the sample you selected.

To do this you simply click and drag your mouse over the text or values in the sample area. This is called painting your fields. Once you have defined the field, you can double click it, right click on it, or click the Field Properties icon to bring up a dialog box that lets you name the field, set its data type, and other options. We’ll delve into the other options in another post.

That’s a fair bit of discussion for one day. If you’re new to Monarch, I hope this helps makes you a bit more comfortable and that you’re that much closer to thinking “I excel with Monarch!”