XML Schematron support in Validibot : An Example from the World of Metrology.

Validating XML with Schematron

As you probably know, there are a few different approaches to checking your XML documents. Schema formats like XSD, Relax NG, or DTD all exist to make sure your XML has the right elements, attributes, types, and structure.

If you're using Validibot, setting up one of these checks is as simple as using the XmlSchemaValidator in one of your workflow steps.

But have another sip of that espresso, because even though your XML doc passes your basic schema checks with ease, it might still be wrong for the business process it represents. That's no good.

This is where schematron comes in. Schematron is a rule-based validation language used to define and check constraints for XML documents.¹ Unlike grammar-based schema languages like XSD or RelaxNG, schematron can analyze the relationships between data points against complex requirements. It helps you check the meaning of your data, as defined by the rules you care about. ¹ Schematron was developed by Rick Jelliffe while he was at Academia Sinica Computing Centre in Taiwan, and later standardized as ISO/IEC 19757-3. In common implementations, a .sch file is compiled to XSLT and produces an SVRL report listing which assertions failed.

If you're a creative thinker, you quickly realize that these two approaches can work well together in one workflow: use XSD to create a simple and clear schema to enforce the document's structure, and then use schematron for the more complex domain rules, calculations, and cross-field relationships that your simpler grammar schema just can't express.

Schematron validations in Validibot

With the release of the new SchematronValidator, these kind of operations are now possible in Validibot. You can build a workflow that not only uses an XSD document in a first validation step to catch silly issues in the XML grammar, but you also add a separate schematron validation step for a rule-based validation, using XPath assertions to check relationships inside the XML.

But that's a bit of hand waving, right? Let's show how this works with an actual example in Validibot. ^{2 2} Validibot community version is open-source and available on GitHub here. There's a quick ref doc for getting set up locally... I tried to make it as simple as possible but perhaps it could be better. Let me know if you have a go and run into any problems.

In this example we'll be using a scenario from the world of manufacturing and measurement device calibration, where people tend to spend a lot of time worrying about measuring stuff. This domain is referred to as metrology and the detailed documents used in manufacturing are called "calibration records."

The focus on precision is quite reasonable: they've got all kinds of gauges, calipers, torque tools, pressure sensors, scales, and other instruments to worry about, and those instruments drift, wear, get damaged, or fall out of tolerance. They use these calibration records to help prove a tool was checked against a known reference, within tolerance, at a particular time.

So where does XML come in? In the real world, calibration certificates may arrive as PDFs, paper records, lab-system exports, or structured data files. But ideally it's the latter: structured data, usually XML. And that's good, because with structured data one can more easily validate, automate and integrate it into other software tools.

Manufacturing people have developed standards to shape these formats, such as the Digital Calibration Certificate, or DCC. The DCC standard is developed by groups in the industry, like Beamex and Physikalisch-Technische Bundesanstalt, to promote a globally recognized XML format for describing calibration information. ³ The PTB DCC site describes DCCs as "a secure electronic document that verifies the performance of a measuring instrument against a known standard," while Beamex calls it the "mp3 of metrology,"⁴ which has a bit more zing. ³ A big part of DCC is the cryptographic signatures used to ensure security and trust. We won't get into this side of things here, but it ⁴ Beamex has a good "what it is and why you should care" article on DCC.

Since DCCs are just structured XML, let's see how we can set up the XSD and schematron validator pair we were just discussing, such that we can make sure some rando supplier's DCC doc is up to snuff.

Now, on to our example. It all starts at your short but legendary stint at Scientific Stuff Manufacturing.

Life at Scientific Stuff Manufacturing

Life is funny, and you somehow find yourself working as a metrologist at Scientific Stuff Manufacturing. Today, a supplier has sent your busy team 100 DCC XML docs for a series of pressure gauges. Each of those docs might include all the fields you expect: issuer, customer, instrument, calibration date, measurement points, uncertainty values, and pass/fail verdicts.

Or...maybe they don't. Maybe the supplier constructed some wrong, or maybe they don't care about fields you do. Above that, your team gets lots of these docs from other manufacturers too, and they keep bugging you about issues with them. Naturally, your first thought would be "ok fine, I can just make an XSD document that verifies the fields exist and that dates look like dates and decimal values look like decimal values and I can go home on time."

But XSD is not the right tool for questions like these:

• Is the certificate issue date after the calibration date?
• If the calibration is marked accredited, did the issuer include an accreditation identifier?
• Does the result unit match the instrument unit?
• Does expanded uncertainty equal standard uncertainty times the coverage factor?
• Does the pass/fail verdict match the tolerance calculation?

Those are cross-field rules, calculations, and policy checks. They are exactly the kind of rules schematron was built to express.

Schematron is the layer that checks whether the certificate's claims are internally coherent. It can verify that the issue date is not before the calibration date, that an accredited calibration includes an accreditation identifier, that result units match the instrument units, that nominal measurement points stay inside the instrument range, and so on.

So you create a schematron file to do just that. But now you've got to code up some custom tool, and share it with your team, and maybe they're not sure how to run it...

Dang, wouldn't it be cool if you had a no-code tool that let you quickly build a workflow that includes your XSD, includes your schematron, logs the validation for analytics and auditing, and then you could just point your team at it and say "if you get any DCCs, run them through this workflow and make sure they're correct before you bug me about them." And as a bonus a valid doc would get a verifiable credential, in case anyone ever asks for proof.

Validibot can help you do just that!

Validibot helps you out

Validibot already includes the XMLSchemaValidator, which can validate XML submissions against XSD, DTD, or RelaxNG schemas supplied by the workflow author. That is the structural layer.

Now, with the new SchematronValidator, Validibot can run your second, schematron-based layer too. A workflow author can upload Schematron rules, and give workflow users results that include rule-level findings such as CAL-MATH-002 or CAL-VERDICT-001. Behind the scenes, schematron rules are compiled to XSLT and executed in an isolated validator backend.^{5 5} Schematron execution is deliberately treated like executable rule evaluation, not like a harmless string check. The workflow author supplies the .sch; on every XML doc submission, Validibot runs the XML through the schematron backend and reports the SVRL-derived findings back to the user. The workflow author writes the .sch rules; Validibot handles the execution path.

A practical workflow looks like this:

1. Run the XMLSchemaValidator against calibration-certificate-demo.xsd.
2. Run the SchematronValidator against calibration-rules-demo.sch.
3. Fail the run when schematron reports fatal findings.
4. Optionally issue a signed validation credential saying this exact document hash passed this exact validation profile.

Setting this up in Validibot is as simple as creating a workflow and then adding two steps, one with XMLSchemaValidator and one with SchematronValidator. Here's what your workflow will look like once that's done:

So what XSD and SCH files did we use for these?

The full files for our example live in the open-source Validibot repo on GitHub (where they double as test fixtures to keep me honest).

• calibration-certificate-demo.xsd
• calibration-rules-demo.sch

With the workflow set up, we're now ready to process some example DCCs sent to your team. Here's two, one valid, one invalid.

• calibration-certificate-valid.xml
• calibration-certificate-invalid.xml

Of course, you are not going to waste your time doing the validation. You're going to point your team to the workflow's upload screen and let them validate the DCCs as they arrive.

When the workflow is run, the XSD step will apply the basic 'shape' rules for the XML. For example, a measurement point must have decimal values for nominal, indicated, correction, standardUncertainty, expandedUncertainty, and tolerance. It also restricts verdicts to pass, fail, or review.

Then, the schematron step applies the more complex rules that check multiple fields at once. For example, this rule checks the date relationship and the accreditation identifier:

<rule context="cal:CalibrationCertificate">
  <assert id="CAL-DATE-001" flag="fatal"
    test="xs:date(@issueDate) ge xs:date(cal:calibration/@performedOn)">
    The certificate issue date must not be before the calibration date.
  </assert>

  <assert id="CAL-ACCRED-001" flag="fatal"
    test="not(cal:calibration/@accredited = 'true') or string-length(normalize-space(cal:issuer/@accreditationId)) gt 0">
    Accredited calibrations must include an issuer accreditation identifier.
  </assert>
</rule>

And this rule checks the calibration math:

<rule context="cal:point">
  <assert id="CAL-MATH-001" flag="fatal"
    test="abs(xs:decimal(@correction) - (xs:decimal(@nominal) - xs:decimal(@indicated))) le 0.000001">
    Correction must equal nominal value minus indicated value.
  </assert>

  <assert id="CAL-MATH-002" flag="fatal"
    test="abs(xs:decimal(@expandedUncertainty) - (xs:decimal(@standardUncertainty) * xs:decimal(../@coverageFactor))) le 0.000001">
    Expanded uncertainty must equal standard uncertainty times coverage factor.
  </assert>
</rule>

That is the main point: XSD can tell you that expandedUncertainty exists and is a decimal. Schematron can tell you whether it agrees with the standard uncertainty and coverage factor.

What the invalid file demonstrates

The invalid example is still shaped like a calibration certificate. It should pass the schema layer. But the business data is wrong in several ways: the issue date is before the calibration date, the accredited issuer has no accreditation ID, the result unit is psi while the instrument unit is MPa, one point is outside the instrument range, and the pass verdict does not match the tolerance calculation.

That is exactly the failure mode this two-layer workflow is meant to catch. A system that only runs XSD validation can say, "Yes, this is a calibration certificate." A system that also runs schematron can say, "Yes, but this certificate should not be accepted."

So while you're blissfully unaware, your team is working through a (nicely formatted and logged) series of issues.

A valid XML document deserves creds

When an XML document passes, your team gets a nice success screen. Here's what we see when we send in our valid XML example:

Like all workflow runs, this one is stored in your Validation Runs log in case you need to remember something about it later.

Hmm. This is all great. But maybe you want to prove to someone else the DCC was validated...and maybe you don't want to have to share the actual DCC that was submitted, just the 'proof'. What then? This is where verifiable credentials come in.