From the Canyon Edge -- :-Dustin

Saturday, January 16, 2021

Awesome, TX BBQ

I've been asked several times over the last few months to share my smoked brisket recipe.  Of course, I'm always happy to, but it's more of a "methodology" than a simple recipe.  In any case, I'm happy to here you go!



Over the years, I've smoked brisket on several different types of smokers.   I started on a Weber bullet smoker, "Smokey Mountain".  For many years, I used an offset smoker, until the firebox eventually rusted through (it was merely 1/16" steel, you really need 3/8" if you want something that's going to last...)

At this point, if I'm smoking a single brisket for my family and neighbors, I'm absolutely loving the Blaze Kamado, from the good folks over at  20" diameter and made of 2.5" thick cast aluminum, this is a Japanese kamado cooker, in the style of a Big Green Egg or Kamado Joe.  But rather than a fragile, ceramic shell, we're talking 200 lbs of solid aluminum.  This thing holds heat beautifully, looks absolutely stunning, and your grandkids will be fighting over it long after you're dead.  I have an unreasonable love for this big, beautiful hunk of metal.


So that's my smoker...  Here's a handful of accessories that I use every time I cook:

  • Drip pan.  The most important piece of your kamado is almost always sold separately.  You absoultely must have a drip pan, to catch grease and maintain humidity inside the smoker.  This is how you'll ensure your briskets are always moist, delicious, and never dry.  I use this 15"x3" cake pan, and it's great.  It fits my Blaze kamado perfectly.
  • Heat deflector plate.  I use a heavy duty steel heat deflector, under the drip pan.  It's a good investment, as it protects your drip pan and prolongs its life pretty much indefinitely.  It also doubles as an awesome pizza-cooking surface.
  • Meater meat thermometer.  I've gone through a bunch of meat thermometers over the years, but my favorite, by far is the Meater.  Bluetooth, with a really clean, intuitive user interface, this thermometer satisfies my inner data nerd while I'm cooking.
  • Grate lifter hooks.  I use two of these for lifting and lowering the grates into place.
  • Ash tool poker.  I use this tool for adjusting the coals before cooking, and cleaning up after.
  • Grill scraper and brush.  I use this brush for keeping my grate clean between cooks.
  • Extra long tongs.  For way too long, I messed around with kitchen tongs.  Don't make that mistake.  Get a set of big, long, heavy-duty tongs.  Useful for moving around coals, or carefully lifting the meat.
  • Chimney starter.  I use a chimney starter to kick start the fire.
  • Fire starter bundles.  Never, ever, ever use lighter fluid.  Kickstart the fire with these bundles.  Or even just a little butcher paper, coated in oil.
  • Lighter.  I'm a dork, so I like this little USB-rechargeable, electric firestarter.  It has a Star Wars feel to it, and it makes me happy every time I use it :-)
  • USB Fan.  I use a little USB fan for starting the fire, and in the rare cases that I need to raise the temperature in the smoker.
  • Lump charcoal.  I use oak lump charcoal.  You might prefer mesquite or hickory or cherry wood or apple wood or pecan.
  • Butcher block.  I do the vast majority of my prep, and even some of the intra-cooking steps (wrapping, flipping) on a 24"x36" butcher block.
  • Stainless Steel Prep Table.  That butcher block sits on top of this prep table.
  • Caster wheels.  And I modified that prep table to have caster wheels, so that I can roll it around easily.
  • Butcher paper and dispenser.  Mounted on that butcher block, I have a butcher paper dispenser and roll of paper.
  • Welding gloves.  I use super heavy duty welding gloves, for any of the hot stuff that I need to touch.  Don't burn your hands!
  • Bread knife.  We'll get to the slicing part later, but get yourself an awesome bread knife.  I'm a big fan of the Miyabi series, but if that's too pricey ($300), here's a great knife for $30.



I can't stress this enough...  Get the best quality meat your local butcher has to offer.  USDA beef is usually graded in three different categories: Prime is generally best, followed by Choice, and then Select.  In fact, some beef has no rating whatsoever.  If you're unsure, you can't go wrong wtih Prime.  I generally avoid Choice and Select, when there are other options.  Once you know what you're doing, or you know what you're looking for (or you have a great butcher who's willing to offer you some help), you can also find some outstanding beef that's ungraded (that doesn't necessarily mean that it's less than Select, just that it's not graded).

You can also get grass fed, or corn fed beef.  I very much prefer the flavors of grass fed, free range cattle.  It's a subtle difference in tastes that you may or may not be able to detect.  But I always find the grass fed beef to be a little earthier, with more umami and richness.  Free range is important to me, as I do want the animals I eat to be treated with respect, and to only ever have one bad day in their entire life.


You can usually buy a whole brisket, or a half brisket.  A whole brisket is two different muscles, called "the flat" and "the point".  The point is the juicy, fattier, more moist piece, and the flat is the leaner, thinner muscle.  I almost always cook a whole brisket.  If I'm going to put 12+ hours into cooking, I want both.  I suppose you could do one or the other, if you're just cooking for yourself or a very small group, but I usually want leftovers, for myself and to give away.

The brisket is underside of the front shoulders of the cow.  Some people are super opinionated about the right or left brisket being better than the other.  I seriously can't tell the difference, so you'll get no opinion here from me.


I generally look for about an 11-12 lbs brisket.  There are certainly bigger, and smaller cuts, but this size seems to hit the right sweet spot for me.  It fits nicely within my 20" kamado, provides 10-15 servings, and has a ton of flavor.  Speaking of servings, you can roughly expect one cooked serving out of every pound of raw brisket.


Trimming a brisket is one my most enjoyable, relaxing, artistic activities in the whole process.  But, I have to warn you...  It can take a long time.  It can be difficult to get right.  You'll probably end up with 2-3 lbs of beef fat rotting in your garbage can (unless you melt it down and make tallow, which is awesome by the way).  Honestly, my advice if you're just getting started, is to skip the trimming step and get a "trimmed" brisket from the butcher.  If your butcher is any good, they'll do a much better job than you (at least at first).

If you are going to trim it yourself, start with a good knife.  A boning knife works pretty well.  I personally use this Miyabi prep knife.  I trim a lot off.  Probably more than one should.  I feel around all of the white fat, and carve off anything that looks unsightly or feels funny.  I want about 3/4" of fat at the thickest parts across the bottom, and splotchy bits around the top and sides (but not big solid chunks).


Nowadays, my seasoning is super simple, and inspired 100% by Aaron Franklin's book and masterclass.  Previously, I used all sorts of salts and peppers and sugars and herbs.

Then, one day, Mr. Franklin taught me the magic of coarse kosher salt and 16-mesh black pepper, in a 50/50 (by volume) mix.  The results are unmistakably amazing.

I start by coating the entire brisket with a little bit of simple olive oil (doesn't have to be expensive or fancy, just organic).  The smoke point of olive oil is low, but we won't be cooking it very hot, so it's great.  Pour a couple of tablespoons on each sides and rub it in with your hands.  You just want a nice, wet, sticky surface.

Then, I combine the kosher salt and 16-mesh pepper in a shaker and absolutely coat every square inch of the brisket liberally.  I haven't been able to overdo it.  Cake it on there.  The magic of the big pepper and salt chunks, is in the surface area.  All of those granules increase the surface area and capture more smoke and flavor.

Do this, and you're going to have a fantastic crust.

Wood & Charcoal

I have the good fortune of living on a couple of acres just in the hills over Austin, Texas, and I have a ton of red oak.  I keep a fire wood rack of red oak slowly aging, seasoning, drying out a little bit.

When I'm able to use an offset smoker, I cook exclusively with wood (no charcoal).  But for this article, let's talk about the kamado...

There's one huge challenge with kamado cooking...  There's really no way to add wood or charcoal during the cook.  You basically have to start with all of the fuel that you're going to use for the whole cook, and it has to last for the whole cook.  I used to really fret about this.  But as it turns out, it's actually pretty straight forward.

So I start with 2 of the little fire starter bundles, at the bottom, and then my chimney smoker loaded with lump charcoal.  Around the outside of the chimney, I usually put 2 or 3 pieces of well seasoned red oak.  Then I light the bales and turn on the USB fan.  It takes about 15 minutes, but as soon as the flames reach the top of the chimney, I dump the coals on the red oak.

And then I basically fill the bottom chamber of the kamado all the way up to the middle rack, with fresh, unlit lump charcoal, and I turn off the USB fan (and put it away).  I find that about 5" of depth of lump charcols, plus 2-3 pieces of oak, will last at least 12 hours, and that generally works for me.  If I ever need to raise the heat, I'll use the fan and blow some extra air through the bottom intake, but this is pretty rare.

Once the coal bed is fully fueled and in good shape, add the second grate, the heat deflector plate, and the drip pan.  I've tried all sorts of things in the drip pan (beer, wine, vinegar, fruits, spices, etc.).  None of it makes any noticeable difference, as far as I can tell.  So I'm back to just plain water and it works great.  The 15"x3" pan that I use holds almost 2 gallons of water, and that lasts about 8 hours in my experience.  I almost always have to top it up once, during the cook, usually at the point that I wrap (175F internal temperature).

Above the drip pan, I put the third grate (the cooking one), and the brisket on that.  I like to start with the fat cap down.  I find this lets the fat cap absorb the earliest heat, and smoke, and provide a little insulation to the rest of the meat for the earliest part of the cook.


I usually aim for a smoker temperature between 220F and 270F.  I'm actually okay with up to 300F (just knowing that it's going to cook a little faster, and will almost certainly need some extra water in the drip pan, because it's boiling off much quicker).

I track the temperature on my phone through the Meater thermometer, and let it get through "the stall", which is a point that every brisket goes through, around 160-170F.  This is a scary point, where you might think your brisket has "stalled" cooking.  Actually, what's happening is that the meat itself is "sweating" water, which is cooling it down, internally, almost as fast as it's cooking.  Resist the urge to mess with it, and let it get through the stall, and once it hits 170F, it'll start climbing again.


At 175F internal, I pull the brisket off and wrap it tightly in butcher paper.  I prefer butcher paper to foil in that butcher paper lets some smoke continue permeating through, whereas foil basically seals off the meat from the smoke.

I wrap it as tightly as I possibly can -- this is always a game to me, trying to beat my previous attempts.  When I put it back on the grill, I place it with the fat cap up.  I find in the last part of the cook (probably the next 2-4 hours), the fat itself melts (ie, renders), and I want all of that beautiful fat to work its way down through the meat.  This is subtle, but I've found this to help me cook some of the visually appealing and delicious briskets I've ever made.


In the unfortunate event that you run out of fuel (heat) before the brisket is cooked, it's really not the end of the world.  You almost certainly got a good 4-6 hours of smoke, at which point, just wrap the brisket in foil, and put it on your gas grill (or even in your oven), at a target cooking temperature of 220F-270F, and cook until done (203F internal temperature).  You won't be able to put a drip pan under it (you shouldn't need one, with it wrapped in foil).  But you should put some sort of a container full of water, to keep it nice and humid in your grill or oven.

It probably won't be your best brisket, but it's certainly not going to be inedible!


Sometimes this is the hardest part, but I like to let the brisket rest, while still wrapped, for at least 30 minutes, and up to an hour.  I usually bring it inside, as all of those aromas can attract flies.  If you have to set it aside for longer than that (maybe it's done, but your company won't arrive for another few hours), then I would wrap the whole brisket (butcher paper and all) with foil, and put it in your oven at the lowest temperature (mine goes down to 175F).


I don't remember...I might have said somewhere else in this post that something else was my favorite part.  Ignore that.  Slicing the brisket is my favorite part.  You can search YouTube for plenty of simple videos on how to slice a brisket.  Maybe I should create my own...

In any case, here's a couple of simple tips.  Start with a good knife.  A 9" bread knife is ideal  I suppose you can use a chef's knife.  And I've seen people use an electric knife (definitely not me...seriously?).  But honestly, please just get a bread knife.  It'll slice a good brisket like butter. You won't be sorry.

I like to slice it with the fat cap down.  (You might need to keep track of this, when you take it off the grill).  There will be a taller, thicker part (the point), and a thinner, smaller part (the flat).  Set the brisket long ways, with the flat farthest away from your body.  Start slicing it at the flat, cutting across the grain (extremely important!), and slowly work your way to the point.

Your first few cuts will be small, less fatty, but each cut will get bigger, and juicier, until you get to the middle of the brisket.  Here, you'll see a marked difference in the meat.  Each slice will include pieces of both muscles, with a thin layer of fat in between.  These are the best slices, in my opinion.

Some people slice the brisket itself in half, laterally, and then carve the point and the flat separate.  I really don't like this.  I love the slices that include some of the point and some of the flat, both.

By the time you get to the very back of the point (the part closest to you), it'll be really moist, and fatty, and just fall apart.  I like to chop this part (the last pound or two, maybe the last 10% of the meat).  These bits make the best chopped brisket sandwiches, and frankly, I think make the best leftovers.


So my other, other favorite part of course is serving.  I love slicing it right off of the butcher block and putting it on someone's plate.  So rewarding, as a pitmaster, to see someone's reaction to your work.  It's just not the same, if I carve the whole thing, and leave it on the counter for my friends and guests to pick their way through.

Okay, so I think that's about it...  I hope this helps someone out there!

See ya!


Thursday, April 2, 2020

How We've Adapted Ubuntu's Time-based Release Cycles to Fintech and Software-as-a-Service

The Spring launch of the Apex 20a platform on March 26, 2020, marks the beginning of an exciting, new era of product development at Apex Clearing.  We have adopted a number of product management best practices from the software industry and adapted them to address some of the unique challenges within financial services and software-as-a-service.  In the interest of transparency, I’m pleased to share our new processes, what we’ve learned along the way, and where we’re headed next. 

Elsewhere in the Software Industry 

I joined Apex Clearing last year, having spent the previous 20 years as a software engineer, product manager, and executive, mostly around open source software, including UbuntuOpenStack, and Kubernetes.  Albeit IBM, Canonical and Google differ from fintech on many levels, these operating systems and cloud infrastructure technology platforms share a number of similarities with Apex's software-as-a-service platform.  Moreover, there also exists some literal overlap: we’re heavy users of both Ubuntu and Kubernetes here at Apex. 

Ubuntu, OpenStack, and Kubernetes all share similar, predictable, time-based release cycles.  Ubuntu has released every April and October, since October of 2004 – that's 32 major software platform releases, on time, every time, over 16 years.  Ubuntu has set the bar for velocity, quality, and predictability in the open source world.  OpenStack’s development processes have largely mirrored Ubuntu’s, with many of the early project leaders having been ex-Ubuntu engineers and managers.  OpenStack, too, has utilized a 6-month development cycle, since 2010, now on its 20th release.  Kubernetes came along in 2014, and sought to increase the pace a bit, with quarterly release cycles.  Kubernetes is a little bit looser with dates than Ubuntu or OpenStack, but has generally cranked out 4 quality releases per year, over the last 6 years.  I’ve been involved in each of these projects at some level, and I’ve thoroughly enjoyed coaching a number of early stage start-ups on how to apply these principles to their product development methodologies. 

Across the board, users, and especially enterprise customers, appreciate the predictability of these release cycles.  Corporate IT managers are better able to plan, test, and roll out technology changes to their environments, with less risk and disruption.  From a commercial perspective, these methodologies drove many wins against legacy, less predictable platforms.  

The Key Principles of Coordinated Cycles 

As a product development team, you have: 
  1. Time 
  2. Work to complete 
  3. Resources to perform work 
To succeed, you must ensure that two of these three are fixed; and only one may vary.  In the Ubuntu methodology – time and resources are assumed to be fixed.  Time is fixed, in that the product will release, on time.  Resources are fixed, in that it takes many months to recruit and hire and on-board new resources.  We hire new people, of course, but we’ll assume those resources will be productive in a subsequent cycle.  Hence, it’s the amount of work that we will complete, which varies.  We will drop or defer commitments, but we won’t change the release dates, and we won't assume additional resources will have meaningful impact within a cycle. 

Within Ubuntu, OpenStack, and Kubernetes, each cycle would “kick-off” with a summit or conference, that brought together hundreds of developers and leaders from around the industry, to discuss and debate designs for the release.  Anyone who’s participated in an Ubuntu Developer Summit, an OpenStack Design Summit, or a KubeCon Design Summit can tell you how essential these gatherings are, to the success of the project.  Within Canonical, we also held a “Mid-Cycle Summit”, exactly at the half-way point of the cycle.  We used this checkpoint, as product and engineering teams, to right-size the scope, and ensure that we hit our release dates, and with the highest quality standards.  Inevitably, new requirements and priorities would emerge, or some committed work proved more complicated than anticipated.  This checkpoint was critical to the success of each launch, as we adjusted the targeted scope for the remainder of the release.

Adapting these Processes to Apex 

When I arrived at Apex in September 2019 to lead the product organization, I inherited an excellent team of product and project managers, peered with high-quality engineering teams.  Products and projects, however, were managed pretty asynchronously, hence release timelines and new feature commitments were unpredictable.  Of course, I had seen this before at a number of the start-ups that I’ve advised, so the model was quite familiar. 

Launch Cycles 

When adapting the coordinated, time-based release cycle to a given organization, the first thing to consider is the time frame.  After talking to all of our stakeholders, 6-month releases, like Ubuntu or OpenStack felt a little too long, a little too slow, for Apex and our customers.  Most of the engineering teams were already quite agile and utilizing 2-week sprints, so getting product requirements for 26-weeks (13 sprints) seemed a little unwieldy.  Quarterly cycles, however, can be pretty tough to see through, for anything but the smallest individual projects (frankly, Kubernetes struggles with the pace at times).  Moreover, all of the projects I’ve been involved in, have struggled with the end of year holidays, in November, December, and January.  Thus, we actually settled on a 16-week cycle, which amounts to roughly 4-month cycles.  That translates to 3 full cycles per year, with 48 weeks of development, while still allowing for 4 weeks of holidays. 

Our cycles are named for the year in which it will complete (launch), and with a letter as an iterator.  In 2020, we launch Apex 20a (March), Apex 20b (July), and Apex 20c (November), and looking forward to 2021, we should see Apex 21a (March), Apex 21b (July), and Apex 21c (November).  The “c” cycles are a few extra weeks, to account for the holidays near the end of the year.  These aren’t really “versions”, as Apex is more like “software as a service”, rather than “delivered software”, like Ubuntu, OpenStack, and Kubernetes.  Also, conversationally, we're referring to the cycles with the season -- so Apex 20a is our "Spring" launch, 20b will be our "Summer" launch, and 20c will be our "Autumn" launch.


Each cycle involves 3 key summits.  As much as possible, these summits are in-person meetings (at least until our travel paused along with the rest of the world).  At this point, we’re proceeding quite seamlessly with virtual summits, instead.  Our summits are recorded in Zoom, and we always take extraordinarily detailed notes in internally shared documents. 
  1. Prioritization Summit 
  2. Planning Summit 
  3. Mid-cycle Summit 


The Prioritization Summit brings together our product managers with all of our key stakeholders – sales executives working with new business prospects, our client relationship managers working with existing customers, as well as our own IT, operations, and site reliability engineers tasked with keeping Apex running on a day to day basis.  Each product manager works with their stakeholders to gather CUJs (critical user journeys) and map those into patterns of similar, weighted product requests. Product Managers generally spend about 2 weeks on that work, which culminates in a session where the Product Manager presents the consensus priorities for their product area for review by the broader product team.  Based on this work, each product manager then starts working on their PRDs (product requirements documents), for the next 3-4 weeks.  Our Prioritization Summit is about a dozen, hour-long sessions, spread over three days in the same week.  We exit the Prioritization Summit with clear stakeholder consensus on stack-ranked priorities for each product family. 


The Planning Summit signals the end of the PRD-writing period, during which product managers worked closely with their engineering counterparts, digesting all of those product requests and priorities, and turn those into product requirements written in RFC2119-style language (must, should, may, etc.).  At the end of that process, each Product Manager and their technical counterpart lead an hour-long session with their plan for the next cycle, including fairly detailed commitments as to the major changes we should expect to be delivered.  Our Planning Summit is about a dozen, hour long sessions, spread over three days in the same week.  We exit the Planning Summit with clear product and engineering consensus on work commitments across the product portfolio, for the upcoming cycle.  This marks the beginning of the development portion of the cycle. 

For the next few weeks, product managers spend the majority of their time with Apex customers and prospects.  Each of us on the product team carry specific OKRs (objectives and key results), to spend meaningful time with our existing correspondents and prospects, communicating our product roadmaps and gathering feedback on their experiences.  We take detailed notes, and all of this data filters directly into our future Prioritization Summits. 


At the middle of our release cycle (week 8 of 16), we bring together the same Product Managers and technical leads to report on status of the first 8 weeks (4 sprints), and recalibrate the remaining work for the cycle.  Without exception, there are always new, late-breaking product requests or requirements that emerge, after the prioritization and planning summits.  Some of these are urgent, and we must accommodate, which usually means something else gets deferred to the next cycle.  Sometimes, we were a little too optimistic with our work estimates, and again we need to adjust.  Occasionally, we’re ahead of schedule and we can cherry-pick some other bite-sized items to bring into scope.  In any case, we will exit the Mid-cycle Summit with a very clear line-of-sight on our deliverables by the end of the cycle. 

With any scope adjustments well understood, the product team shifts into “go-to-market" mode.  Over these next 3 weeks, Product Managers are working with our Marketing counterparts, writing release notes, creating marketing content, educating our sales teams, and working through signoffs on our launch checklists. 

At this point, the cycle repeats itself.  Once our go-to-market activities are complete, Product Managers shift back into prioritization mode, working with our stakeholders, while the engineering team completes their work and the marketing team publishes the launch. 

Speaking of...let’s talk about the Apex 20a Release. 

The Apex 20a and 20b Releases 

Apex 20a launched on March 31, 2020, as our first release using the methodologies described above.  Apex clients can find detailed release notes in the Apex Developer Portal.  This cycle began with a Prioritization Summit in October 2019, a Planning Summit in November 2019, and a Mid-cycle Summit in January 2020.   This cycle involved 17-weeks of development. 

Our work on Apex 20b is already well underway, having held our Prioritization Summit in February 2020, and we’re holding our Planning Summit this week (March 2020).  Our Mid-cycle Summit will be held in May 2020, and we will launch Apex 20b in July 2020. 

It’s important to note that although we do have a very specific “launch date”, which signals the end of the development cycle, each of our engineering teams have developed, tested, and deployed to production hundreds of changesets during the cycle.  Thus, we maintain our agile CI/CD (continuous integration / continuous deployment) systems within every product and engineering team.  To be clear, we don’t “hold” anything specifically until launch date.  This is a very specific differentiation from Ubuntu, OpenStack, and Kubernetes, which are “shipped software”, as opposed to Apex technologies, which amount to “software as a service”.  For these reasons, we try to use the term “launch”, rather than “release”, when we talk about the “launch date” at the end of the cycle.  All, that said, we have found the processes described here very useful in our planning and communications about Apex technology to our customers. 

In Conclusion 

Apex 20a is the first of many coordinated product launch cycles our customers will experience.  We’ve adapted many of the best practices utilized by the open source software industry as well as Silicon Valley, and those practices are helping us work more effectively with our tech-savvy client base.  Apex will have 3 launches in 2020 (20a, 20b, 20c), and at least 3 launches in 2021.  By openly sharing our product stages and delivering a consistent, predictable, and reliable schedule, there are now ample opportunities for both customer input and detailed review and oversight by our leaders, which culminates in secure and stable products for our industry.  We’re delighted at the engagement thus far, and really look forward to more collaboration in the future. 

On behalf of the Apex product team,