Padraic Renaghan

PostgreSQL Export ntile() bucketing

Tue, 19 Mar 2024 11:00:02 -0400

This post concludes recent work and commentary on exporting masked data from one AWS Aurora PostgreSQL environment to another:

We were able to export masked data successfully and import it without issue. The export is significantly slower than the import, at least in our case where we needed many of the exports to have joins in order to restrict what data was exported.

AWS RDS Aurora PostgreSQL has special features to contain the low memory situation and cancel the offending query rather than have the server hit OOM and crash, but the problem of being unable to export the data remains.

In order to avoid the dreaded OOM killer that can occur when an OLAP style query runs awry, it is important to break up large exports into smaller parts. Useful also to run them in parallel.

The PostgreSQL ntile() function is useful here. Basically given a table and a set of columns (ideally a unique index), the ntile function creates a (provided) number of buckets of rows on the table. Each bucket of roughly equal size. The column values for the columns are provided so that you can then query for the range of rows in each bucket.

For example, given this simple two column table with a primary key and some sample data:

CREATE TABLE mydata(
  col1 INTEGER NOT NULL,
  col2 INTEGER NOT NULL,
  PRIMARY KEY(col1, col2));

INSERT INTO mydata(col1,col2) VALUES 
(1,1) ,(1,2) ,(1,3) ,(1,4) ,(1,5),
(2,50),(2,40),(2,30),(2,20),(2,10),
(3,5) ,(3,4) ,(3,3) ,(3,2) ,(3,1),
(4,99),(4,98),(4,97),(5,1) ,(5,2);

And say you want to break those 15 rows into 5 roughly equally sized buckets:

SELECT ntile(5) 
  OVER (ORDER BY col1,col2) bucket, col1, col2
FROM mydata ORDER BY bucket, col1, col2;

b	c1	c2
-	-	-
1	1	1
1	1	2
1	1	3
1	1	4
2	1	5
2	2	10
2	2	20
2	2	30
3	2	40
3	2	50
3	3	1
3	3	2
4	3	3
4	3	4
4	3	5
4	4	97
5	4	98
5	4	99
5	5	1
5	5	2

This gives you each row from the table with the bucket that it is a part of. One thing to note in this example is that it is important to think of this as a composite key, not 2 independent columns. Note that the start of bucket 5 is (4,98) - if you simplistically take the minimum of each column in the bucket you would get (4,1), which would be incorrect causing buckets 4 and 5 to overlap.

After much trial and error I arrived at this query which gives you the starting composite key for each bucket:

SELECT DISTINCT ON(bucket) bucket, col1, col2
FROM (
     SELECT  col1, col2
            ,ntile(5) OVER (ORDER BY col1,col2) bucket
     FROM mydata
    ) parts
GROUP BY bucket, col1, col2
ORDER BY bucket, col1, col2;

b	c1	c2
-	-	-
1	1	1
2	1	5
3	2	40
4	3	3
5	4	98

Then when exporting you can use PostreSQL ability to create a virtual row for the composite key. For example to select bucket 4:

SELECT col1, col2
FROM mydata
WHERE (col1, col2) >= (3,3)
  AND (col1, col2) <  (4,98);

For the first bucket use < (start of 2nd bucket), for the last bucket use >= (start of last bucket), and for buckets in between use a combination as in the example above.

AWS RDS Aurora PostgreSQL Table Import from S3 Stub

Mon, 05 Feb 2024 12:00:04 -0400

Here is the aws_s3.table_import_from_s3 stub that pairs nicely with the previously blogged aws_s3.query_export_to_s3 stub.

CREATE SCHEMA IF NOT EXISTS aws_s3;
CREATE OR REPLACE
  FUNCTION aws_s3.table_import_from_s3 
    (table_name TEXT, column_list TEXT, opt TEXT,
     bucket TEXT, path TEXT, region TEXT, OUT result TEXT)
AS
$$
DECLARE copy_stmt TEXT;
  rows BIGINT;
BEGIN
  copy_stmt := 'COPY ' || table_name;
  IF column_list IS NOT NULL AND length(column_list) > 0 THEN
    copy_stmt := copy_stmt || ' (' || column_list || ')';
  END IF;
  copy_stmt := copy_stmt || ' FROM ''' || path || 
    ''' WITH ' || opt || ';';
  EXECUTE copy_stmt;
  GET DIAGNOSTICS rows = ROW_COUNT;
  result :=  rows || ' rows imported into relation "' || 
    table_name || '" from file ' || path || 
    ' of ' || rows || ' bytes';
END;
$$
LANGUAGE plpgsql
;

AWS RDS Aurora PostgreSQL Query Export to S3 Stub

Fri, 02 Feb 2024 12:00:04 -0400

AWS RDS and Aurora database services have a very useful enhancement, aws_s3.query_export_to_s3, to the native PostgreSQL COPY command, that lets you export from RDS/Aurora directly to AWS S3 storage. This can be very useful for moving data around, including exporting from Production with masking functions to land in test/debugging environments.

There are currently some limitations around cross-account exporting, like you can’t. But simple workarounds to post-process the files to either rekey them, or copy them yourself exist.

Your local machines won’t have this AWS specific procedure but or local testing you can write a stub and install into your local PostgreSQL database that gives a good approximation of what the AWS procedure will do.

Note: dollar-quoting is highly recommended for wrapping around the select statment. Will make your life much easier if you are programmatically generating these statements and export commands.

Update See related aws_s3.table_import_from_s3 stub.

CREATE SCHEMA IF NOT EXISTS aws_s3;
--
-- For local testing
-- stmt: The select statement to be exported
-- bucket: S3 bucket name, ignored
-- path: Full absolute path to the desired output file
-- region: S3 region, ignored
-- opt: COPY options
-- returns exported rows, files (always 1), and bytes (faked)
--
CREATE OR REPLACE
  FUNCTION aws_s3.query_export_to_s3(
     stmt TEXT, bucket TEXT, path TEXT, 
     region TEXT, opt TEXT, 
     OUT rows bigint, OUT files bigint, OUT bytes bigint)
AS
$$
DECLARE copy_stmt TEXT;
BEGIN
  copy_stmt := 'COPY (' || stmt || 
               ') TO ''' || path || 
               ''' WITH (' || opt || ');';
  EXECUTE copy_stmt;
  GET DIAGNOSTICS rows = ROW_COUNT;
  IF rows > 0 THEN
    files := 1;
    bytes := rows * 1024;
  ELSE
    files := 0;
    bytes := 0;
  END IF;
END;
$$
LANGUAGE plpgsql;

AWS RDS Aurora PostgreSQL Masking Functions

Thu, 01 Feb 2024 12:00:04 -0400

Protecting your data is important, both from people who want to get it, and more importantly from your own staff who are only-human, and thus subject to making mistakes.

Using Production data can be very useful for all kinds of debugging and testing. But you don’t want to expose confidential production data to staff that do not need to see it.

Masking is useful as you export data from Production and load it into your test/debug environment.

Below are a few PostgreSQL functions to get you started…

Thanks to Szymon Lipiński for useful info on this topic.

In general, NULL and zero-length string inputs are returned unchanged, otherwise a randomized string of the same length as the original is returned. PARALLEL RESTRICTED for those that use random(). md5 is used over a single random() call, to get 32 alphanumeric characters from that single random() call.

Update Mar 19 2024: Functions I actually used provided.

-- generate a random number, will be a 0.[16 digits] decimal
-- cast to text, then used as bytea intput to md5 function
-- md5 will output 32 alpha numeric characters
-- do this as many times as you need to get at least the desired length
-- joining here with space in between
-- trim back to desired length at the end
-- we need the greatest wrapper otherwise generate_series returns null
-- which on an empty string causes the output to be null, we need empty
-- string input to be empty string output
CREATE OR REPLACE FUNCTION mask_alpha_num_space(orig TEXT)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT substring(
         (SELECT string_agg(md5(random()::TEXT), ' ')
          FROM generate_series(1, GREATEST(1, GREATEST(1, CEIL(length(orig) / 32.)::integer)))),
         1,
         length(orig));
$$;

-- same as above, but with join of empty string so no spaces
CREATE OR REPLACE FUNCTION mask_alpha_num(orig TEXT)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT substring(
         (SELECT string_agg(md5(random()::TEXT), '')
          FROM generate_series(1, GREATEST(1, CEIL(length(orig) / 32.)::integer))),
         1,
         length(orig));
$$;

-- generate a random number, will be a 0.[16 digits] decimal
-- cast to text, then get the 16 digits after the decimal
-- do again and again if need longer
-- substring at the end to the desired length
CREATE OR REPLACE FUNCTION mask_num(orig TEXT)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT substring(
         (SELECT string_agg(substring(random()::TEXT, 3), '')
          FROM generate_series(1, GREATEST(1, CEIL(length(orig) / 16.)::integer))),
         1,
         length(orig))
$$;

-- same as above but here we have the length desired
CREATE OR REPLACE FUNCTION mask_num(len INTEGER)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT substring(
         (SELECT string_agg(substring(random()::TEXT, 3), '')
          FROM generate_series(1, GREATEST(1, CEIL(len / 16.)::integer))),
         1,
         len)
$$;

-- a random 3 digit + random 10 alphanumeric + ' St'
-- leave the input even though unused so that if the
-- input is null then PG won't bother calling function
CREATE OR REPLACE FUNCTION mask_street(orig TEXT)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT substring(random()::TEXT, 3, 3)
         || ' '
         || substring(md5(random()::TEXT), 1, 10)
         || ' St';
$$;

-- create a base64 encoded sha256 has of the input
CREATE OR REPLACE FUNCTION mask_hash(inp TEXT)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL SAFE
AS
$$
SELECT encode(sha256(convert_to(inp, 'UTF8')), 'base64');
$$;

-- generate a random email address
-- leave the unused input so PG will skip calling function if it is null
CREATE OR REPLACE FUNCTION mask_email(str TEXT)
  RETURNS TEXT
  LANGUAGE SQL
  RETURNS NULL ON NULL INPUT
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT substring(md5(random()::TEXT), 1, 10) || '@example.com';
$$;

-- if the flag is FALSE then return the orig string as is
-- otherwise mask the string
-- this function DOES NOT return null on null input
CREATE OR REPLACE FUNCTION mask_alpha_num_flagged(str TEXT, bool BOOLEAN)
  RETURNS TEXT
  LANGUAGE SQL
  PARALLEL RESTRICTED -- because random
AS
$$
SELECT CASE
         WHEN str IS NULL OR NOT bool THEN
           str
         ELSE
           substring(
             (SELECT string_agg(md5(random()::TEXT), '')
              FROM generate_series(1, GREATEST(1, CEIL(length(str) / 32.)::integer))),
             1,
             length(str))
         END;
$$;

Traveling with your iPhone

Mon, 29 May 2023 11:00:00 -0400

A few notes on traveling with your iPhone, to a foreign country, where you want to be connected for Messages/Phone/Maps, but don’t want to burn through a bunch of data needlessly…

Your modern iPhone likely has support for multiple esims. This makes it easy to keep your normal esim in place, add a separate local/regional esim for traveling data, and have close to the best of both worlds.

All without paying exorbitant International Data Fees to Verizon, AT&T, or T-Mobile.

Airalo is great and comes highly recommended.

Good info here from Apple on getting the esim setup. Airalo setup info here. Be sure and do the preliminary parts at home, on good WIFI, so when you land you can flip the esim on and be up and running.

Install the Airalo App so you can manage your esims, top-up with more data, and monitor your data usage.

Basically you leave Phone, iMessage, FaceTime set to your “primary” SIM (normal provider), and set the Cellular Data to the Airalo SIM.

Make sure you have “WIFI Calling” ON under Cellular. This way you can still have calls using your normal number, while traveling.

Beyond that the key is to conserve data and battery while keeping the phone useful.

Related iPhone settings to adjust:

Settings → Personal Hotspot → Allow others to join → Disable
Settings → TV → Streaming Cellular Data → Disable
Settings → TV → Download Cellular Data → Disable
Settings → Photos → Cellular Data → Disable
Settings → Music → Cellular Data → Disable
Settings → Cellular → Disable cellular data for video apps (Youtube, Hulu, Disney, Netflix, Max, Showtime…)
Settings → Cellular → Disable cellular data for music apps (Spotify…)
Settings → Cellular → Disable cellular data for podcast apps (Overcast…)
Settings → Cellular → Other
Settings → Cellular → Disable Wi-Fi Assist (scroll all the way down)
Settings → Cellular → Disable iCloud Backup (scroll all the way down)
Settings → Display & Brightness → Turn brightness down
Settings → Low Power Mode → Enabled
Settings → Messages → Low Quality Image Mode → Enabled - this is a maybe, we send lots of trip photos to each other, so I don’t normally enable this
Settings → Cellular → eSIMs → select → Data Mode → Low Data Mode
Settings → App Store → Cellular Data → Disable Automatic Downloads
Settings → App Store → Cellular Data → App Downloads → Always Ask
Settings → Translate → Downloaded Languages → Select Languages ↓
Settings → Safari → Reading List → Enable Automatically Save Offline

For Other, the list is sorted by most data usage first, so look at that list and anything that you don’t need while out for a day of traveling, disable. Likely Maps and Safari are up at the top of the list, I recommend you leave those ENABLED. Maps and looking up local information are obviously important. Just don’t get carried away, especially with Safari.

Download podcasts, music, and videos to watch on the plane ahead of time. Streaming video (followed by streaming audio) uses more data then most anything else you’ll ever do on your phone.

Also be sure to pre-download offline maps in Maps app, and foreign languages in Translate app.

Also don’t use Personal Hotspot.

You can do all of this before you leave for your departure airport. Get yourself a backup battery for the family to share - invaluable.

On your trip keep an eye on data usage from the Airalo app, and watch the apps using the most cellular data using the Settings → Cellular screen. Adjust as you go to keep your usage down, your phone useful, and your battery always with a charge.

You cannot control order of Gradle dependencies

Wed, 24 May 2023 11:00:00 -0400

You cannot control the order of Gradle dependencies.

You will read that you can, it’s not true.
You will read that the order you specify the dependencies is the order they are used when resolved, also not true.
You will try some tricks thinking the order is alphabetical, or as specified, both not true.
You will ask about it and get judgy responses about how it is bad practice and you shouldn’t ever need to do that, don’t have the same class in two places… and so on.

This is mostly nonsense covering for a deficiency in Gradle, Java clearly supports the idea of class dependency ordering by supporting an ordered/hierarchical list of classloaders, as well as the JVM classpath having ordered entries.

But you still want to use Gradle, because it is otherwise excellent, so you need a way to move forward…

The answer is to adjust your dependency jars to avoid the conflict. For example if you have a class A.class that is in a third-party dependency jar and a local project override of that same class, and you want to be sure your local override is always used, then do something like this:
(original credit to Lance)

// create a new config, of the jar we want to trim
// no transitive dependencies, only that one jar
val jarTrimConfig: Configuration by configurations.creating
jarTrimConfig.isTransitive = false

dependencies {
  // identify the one jar that needs trimming
  jarTrimConfig("third-party:package:1.0")
  // add the trimmed jar to our classpath
  implementation(files(trimJar))
}
// task to create the trimmed jar
val trimJar = tasks.register<Jar>("trimJar") {
  archiveFileName.set("$buildDir/libs/trimmed-package.jar")
  from(zipTree(jarTrimConfig.singleFile))
  exclude("third-party/package/A.class")
}

Recording Skype (Teams, Zoom, Webex) to MP3 with Audio Hijack

Wed, 08 Jul 2020 20:00:00 -0400

Goal

Record a Mac Skype for Business meeting to MP3 – participating in the meeting as normal with my headset – creating a MP3 with myself and other participants heard equally in both left and right channels.

Took a couple of days, but figured it out. Here is the how-to…

Note: I did this for Skype and Microsoft Teams. But the same approach will work for any conferencing software like Zoom or Webex.

What you need:

The recommended approach from Rogue Amoeba does work, but its problem is the MP3 has my voice on the left channel (ear) and other participants on the right channel. Which, although audible, I find uncomfortable to listen to.

The below solution records yourself and others in mono MP3, at equal volume, in both ears.

How-To:

Install Skype
Install Audio Hijack
Connect your headset
In Skype preferences set all your Audio Video devices to your headset

Create a New Session in Audio Hijack. You can mimic the screenshot below, or download the session file

I am a low-talker, so I have my microphone volume increased to 150%
Make sure for both Skype source elements you open the Advanced settings and disable Include audio input

For the MP3 recorder, the consensus on the Internet for recording voice is
- 96 Kbps bitrate
- constant bitrate CBR
- 44100 Hz sampling rate
- mono channels
With those settings you will generate approximately 1MB in MP3 file size per minute recorded
Adjust File name pattern, Save to output directory, Title, and cover image as needed

In-Action:

Open the session in Audio Hijack, click to start recording
Join your meeting in Skype
You will hear participants in your headset as normal
Your voice and participants will be recorded in mono to the MP3 file
Observe the VU Meters in Audio Hijack and make sure you see appropriate activity
- Top meter, to MP3, triggers on both your voice and others voices
- Bottom meter, to headset, triggers only on others voices
Check that the MP3 file appears and is growing as you record
Click to stop recording
Success

If you need to edit the MP3 after recording I find Fission - Rogue Amoeba $29 to be a great tool for lossless editing/ trimming/ chapters/ splitting/ transcoding… I also played around with Loopback - Rogue Amoeba $99 – neat advanced tool but I did not need it for this task.

fwd:cloudsec Cloud Security Conference

Fri, 26 Jun 2020 12:00:00 -0400

fwd:cloudsec security conference is being held virtually, streaming on twitch, on Monday June 29, 2020 starting at 10:00 am Eastern Time.

Looks to be a great lineup of talks from the experts.

The conference is non-profit, free, and open to all to attend.

I created an iCal fwdcloudsec2020.ics file of the conference schedule for those that want to load into their calendar .

Weather Apps - Dark Sky, Hello Weather, and more

Thu, 16 Jan 2020 09:00:00 -0400

I seem to try, and pay for, almost every new iPhone weather app.

Weather is an essential function of my iPhone, I’d say number 3 after and

Morning check, what the lies ahead today
Making sure kids dressed appropriately for going to school and walking home
Preparation for upcoming storms
Intra-day rain forecast, “if I wait 10 minutes to walk to my car the rain will have stopped”
Weather Push-Alerts, National Weather Service Winter Storm Warnings
Week outlook, “I can mow the grass on Saturday when it is mild and dry”
Watch Complications for temperature and quick forecast on my wrist

I have tried, paid-for/subscribed, and liked

Most of these, if not all, use forecast data from Dark Sky so the differences don’t really have anything to do with the accuracy of the forecast. Mostly it is about user interface, being able to quickly and easily accomplish the above functions. All the while minimizing noise, clutter, and complication.

I keep coming back to Dark Sky.

Dark Sky meets all the feature requirements, has a solid, uncluttered interface, and push notifications. Carrot Weather was close, but I found it had too much information and felt cluttered. Weather Line is nice, and the line metaphor is great, but I found it took too many taps to get the location I wanted to see. Hello Weather is almost perfect, better user interface than Dark Sky, but it lacks advanced features like push notifications.

So back to Dark Sky, but the itch is never scratched, so even now I am experimenting with two apps - Hello Weather for everyday use, Dark Sky kept in background for push notifications. I get the pleasant, straightforward, colorful interface of Hello Weather, with the reliable notifications from Dark Sky.

So far so good.

Related:

Dark Sky on the web - fantastic, I use this frequently
iStat Menus forecast on Mac - I use this
Forecast Bar forecast on Mac
LaunchBar custom action for weather forecasts

Create Text File Here - Mac/Finder/Keyboard Maestro

Thu, 03 Oct 2019 09:00:00 -0400

A Keyboard Maestro macro that creates an empty text file in the currently selected folder in Finder.

Sets the Finder selection to the newly created text-file.txt

Puts Finder in Rename mode on the new file

The Keyboard Maestro macro is shown below as screenshots. Also available to download as Text-File-Here.kmmacros and import into Keyboard Maestro.

Similarly, although Finder already has a New Folder option, here is a Keyboard Maestro macro to download that creates a new folder and puts Finder in rename mode

Update Oct 5: Macro now offers to save text clipboard contents into the new file

Conference Call Headset Jabra Evolve 75

Tue, 06 Aug 2019 09:00:00 -0400

I work from home and my kids think my job is to “talk on the phone”, which tells you how often I am on conference calls

Apple Earbuds and AirPods weren’t doing the trick for a variety of reasons.

As usual, I asked The Wirecutter and purchased per their recommendation.

The Jabra Evolve 75 headset is great

Sound is good
Microphone capture works well
Flick-the-microphone to mute is handy
Connects to multiple devices at once
All day charge
I also purchased the stand which isn’t required but very convenient
I use with one ear fully covered and the other ½ covered (so I can hear ambient noise)
Alternatively they have noise-cancelling
Not cheap, but as the kids note, it’s mostly what I do

Highly Recommended

Outlook for Mac Raw Query based Smart Folder

Sat, 03 Aug 2019 09:00:00 -0400

My recommendation for Mac power-user email client is MailMate. No question. No contest.

However sometimes work security policies force you into Microsoft-Exchange-land and your only choice is to use Outlook for Mac

You won’t ever be able to make Outlook work as well as MailMate, but you can do a few things to help you get closer. MailMate Smart Mailboxes can be somewhat mimicked using Outlook Smart Folder feature along with the Raw Query option.

Outlook mail search is based on the Mac Spotlight search and index system. So make sure you have Spotlight enabled for mail in System Preferences → Spotlight → Mail & Messages:

Then ⌘⌥F to enter Advanced Search mode, then click the Item Contains drop-down and select Raw Query

Now you can enter your raw query in the text box. The raw query is Spotlight query syntax and fields.

To find out what fields Spotlight makes available, run mdls on an Outlook mail message file. In my case Outlook stores mail messages under ~/Library/Group Containers/UBF8T346G9.Office/Outlook/Outlook 15 Profiles/Main Profile/Data/Messages/*/*

$ mdls 8459FCED-2F8B-6A2E-B12E-1BC7C3DFFE33.olk15Message
com_microsoft_outlook_accountID                = 3
com_microsoft_outlook_completed                = 0
com_microsoft_outlook_flagged                  = 0
com_microsoft_outlook_folderID                 = 166
com_microsoft_outlook_forwarded                = 0
com_microsoft_outlook_has_attachments          = 0
com_microsoft_outlook_has_mentioned_me         = 0
com_microsoft_outlook_has_reminder             = 0
com_microsoft_outlook_has_text_content         = 1
com_microsoft_outlook_hasDueDate               = 0
com_microsoft_outlook_hasStartDate             = 0
com_microsoft_outlook_junkLikelihood           = -100
com_microsoft_outlook_messageReceived          = 2019-08-04 19:18:22 +0000
com_microsoft_outlook_messageSent              = 2019-08-04 19:18:19 +0000
com_microsoft_outlook_primaryCategory          = 0
com_microsoft_outlook_priority                 = 0
kMDItemDisplayName                             = "AWS CodeBuild www-site Complete"
...

Use those fields to form your raw query. For example I have a Smart Folder named Todo where I want to include anything in the Later folder folder or that is flagged and that isn’t completed yet and isn’t in the Spam or Trash folder:

  (com_microsoft_outlook_folderID == 123
|| com_microsoft_outlook_folderID == 154
|| com_microsoft_outlook_flagged == 1)
&& com_microsoft_outlook_completed == 0
&& com_microsoft_outlook_folderID != 166
&& com_microsoft_outlook_folderID != 162

You may also find the below Apple Script helpful. Select a folder in the Outlook UI and run the script using Script Editor, it will log the folder id of the currently selected folder.

tell application "Microsoft Outlook"
	get selected folder
end tell

Block Mail Trackers

Fri, 05 Jul 2019 10:30:00 -0400

The recent dust-up over the Superhuman mail application tracking when recipients open email, down to the level of the exact timestamp every time they opened it and their location, brings to light an important privacy issue. When/where/if you read an email someone sent you is private. Email services that attempt to track that are intrusive.

There are a few different ways you can reduce mail tracker spying…

If you use GMail within Chrome or FireFox try Ugly Mail or Pixel Block or Trocker extensions
Set your mail client to not display remote images

Or, more blunt but highly effective, block known mail tracker domains at the network level. Little Snitch on a Mac is a great tool for this…

Little Snitch

Little Snitch is a network connection filter. Allowing you to allow or deny network access on your Mac.

Download and install Little Snitch 4. This installs a kernel extension, so be sure to follow the instructions to allow the extension, and restart your Mac as indicated.

Rules

I compiled a Little Snitch block list based on articles and extensions.

Set up the rules yourself or download and import my list, or subscribe to it.

MacBook Pro Vega, Flicker, Working from an iPad, Drafts

Fri, 12 Apr 2019 09:00:00 -0400

Purchased a new 15" MacBook Pro. So far so good with the keyboard, but had problems with screen flicker. Took the MBP in for Apple Genius service and after 2 separate send-away-for-repair incidents, MBP coming back with no new parts, the local Apple Store gave up and just gave me a new MBP as replacement. Of course I still had the screen flicker with the replacement

Some digging on the Internet led me to the conclusion that the problem is in Mac OSX software, in combination with the new Radeon Pro Vega 20 graphics card, and the fact that I set the screen resolution to “Scaled” (need the bigger text for my aging eyes ). Once I switched back to “Default for display” resolution the flickering stopped.

During the send-away-for-repair incidents I did get a chance to work exclusively from my 11" iPad Pro with Pencil which was an interesting, largely successful, experiment. Below is a rundown of what worked well and not…

The usual suspects, apps I use all the time and continued to use:

Apple Mail (I have tried Spark and others, but keep coming back to old reliable)
Safari
Apple Files - having iCloud Drive for my desktop ~/Documents folder was extremely useful
Tweetbot - Twitter
Unread - RSS reading
Apple Books - EPUB book reading
Things - Todo
Fantastical - Preferred calendar
iA Writer - Text notes, synced to iPhone and Mac via iCloud, great UI, great search

Corporate-wise we are a Microsoft Office 365 shop, so fairly standard footprint of Microsoft apps, all of them work well on iPad

Microsoft Teams - Lessor Slack for O365’ers
Microsoft Skype - Ugh, why is the app I most rely on the worst?
Tip: this Jabra Evolve 75 headset is fantastic
Microsoft Outlook - occasionally needed since work doesn’t let us use Apple native Exchange integration
Microsoft Word - this worked surprisingly well, editing docs, tracking changes, pencil was handy to navigate. Some continued corporate silliness required me to email myself a copy of the Word doc so I could extract a clean copy from the corporate O365 walls

The most significant app turned out to be Drafts. Drafts let me glue everything together. A quick place to jot down a thought. A place to draft an email. A place to start a document. I had only used Drafts very lightly before, but working on the iPad pushed me to always start in Drafts then take the text on to its final home (usually an Apple Mail message, or Microsoft Word document).

#fail

There were three things I could not do on the iPad

Calibre - I use Calibre to manage my EPUB ebook collection, and help me extract “unencumbered” EPUBs from borrowed library Kindle ebooks. This is not possible on iPad
Personal Finance - I could probably setup Moneydance to work somehow with iPad but it was harder, more involved, required Dropbox, than I wanted to bother with
Code Search - I work on a large financial services enterprise Java application. A lot of what I do is look at code, searching code, proposing minor patches to code. Working Copy is a fantastic application, but searching across an entire project is slow and difficult to use. Perhaps it is due to the size of my git repository. But the bottom line is there is nothing remotely equivalent to my desktop IntelliJ IDEA editor with its context aware search capabilities.

Overall I was successful working on the iPad. Hoping someone fills the gap around context aware code search. Working Copy continues to improve, and I look forward to making use of its new features.

Block Messages and FaceTime on Mac - "Homework Computer"

Mon, 25 Mar 2019 09:00:00 -0500

The Mac has very good parental controls, but it isn’t as easy as I’d like to simply block Messages and FaceTime . Hopefully Screen Time will be supported on the Mac someday. For now, if you are trying to setup a “Homework Computer” for the kids, here is a way to block Messages and FaceTime using Little Snitch.

Little Snitch

Little Snitch is a network connection filter. Allowing you to allow or deny network access attempted by particular software on your Mac.

Download and install Little Snitch 4. This installs a kernel extension, so be sure to follow the instructions to allow the extension, and restart your Mac as indicated.

Purchase it ($45 USD) once you get this all working.

Configuration

For a simple “Homework Computer” setup, that the kids can’t mess with, configure in the Little Snitch preferences:

General - Operation Mode select Silent Mode - Allow Connections
Security - disable Allow Rule and Profile Editing
Security - disable Allow Profile Switching
Security - disable Allow Preferences Editing
Security - disable Allow Global Rule Editing
Security - disable Allow GUI Scripting

Rules

The specific rules to block Messages and FaceTime are fairly simple:

Block Incoming and Outgoing network traffic with Apple Push Notification Service /System/Library/PrivateFrameworks/ApplePushService.framework/apsd
Block Incoming and Outgoing network traffic with Messages application
Block Incoming and Outgoing network traffic with FaceTime application

Set up the rules yourself (make them “Global” and “High Priority”), or download and import, or subscribe

Note: the Apple Push Notification Service may stop other push based services on the Mac from working, but I haven’t had any trouble

AWS Lambda boto3 Version - User.Tags missing

Mon, 21 Jan 2019 15:00:00 -0500

Wasted a day chasing down this issue, turns out to be a simple cause. Writing it up here to hopefully save others a few hours…

So you want to retrieve an IAM user’s tags inside a Lambda function. Perhaps in a Lambda triggered from a CodeCommit push, that posts a notification to Slack or Microsoft Teams - and the Tag has the user’s full name and email address you want to use in the notification.

So you setup a new AWS IAM user with tags:

Then a python3 Lambda using boto3 to retrieve the tags:

import boto3

def lambda_handler(event, context):
  iam = boto3.client('iam')
  userName = 'jsmith';

  try:
    response = iam.get_user(UserName=userName)
    user = response['User']
    email = None
    if 'Tags' in user:
      for tag in user['Tags']:
        if tag['Key'] == 'Email':
          email = tag['Value']
      if email == None:
        print('get_user.Tags missing Email tag')
      else:
        print('Found user email tag ' + email)
    else:
      print('get_user response missing Tags')
    
  except Exception as e:
    print('ERROR get_user {}'.format(userName))
    print(e)
    raise e

Run and you get: get_user response missing Tags

Hmm…

our IAM user does have tags
boto3 API documentation says get_user returns Tags
but it is not working

Take the Lambda, turn it into a command-line python script by removing the lambda_handler method, and run locally. You get Found user email tag jsmith@example.com.

So it works locally but does not work inside Lambda.

Surely retrieving Tags on a User has been around forever. So I thrash around for a few hours, thinking I am not calling the method correctly, passing the wrong inputs, not handling the output correctly… nothing works. Then I figure I will just double-check the version of boto3 running everywhere.

boto3 documentation website shows the latest version is 1.9.82
local command line pip3 show boto3 shows 1.9.75
Adding this statement to our Lambda function print('boto3 version: ' + boto3.__version__) and running shows 1.9.42

Was get_user returning User.Tags introduced after 1.9.42? You can look at the documentation for older versions by changing latest in the URL to the version you want, so let’s look at the documentation for 1.9.42 at https://boto3.amazonaws.com/v1/documentation/api/1.9.42/

Boom! , User.Tags is not shown in the response. Version 1.9.42 running in Lambda was released November 9, 2018 and does not return User.Tags.

Presumably AWS will upgrade the Lambda version shortly. Thankfully, I can wait. This blog post has some helpful info if you can’t wait and want to run Lambda with your own version of boto3 (or if you use Javascript Lambda see this post).

Cold Brew Coffee Recipe, New Orleans Style

Tue, 15 Jan 2019 21:00:00 -0400

This is my recipe for New Orleans Style Cold Brew Coffee . Cold-Brew is easy to make in bulk, ahead of time, and is less bitter that traditional methods.

Chicory and Chocolate give it a distinct flavor. Adjust along with vanilla, cinnamon and other add-ins as you wish.

This recipe makes 1 gallon, and takes 24-hours . Every two weeks I make 2 gallons over the course of 2 days.

Beans

I have tried a variety of beans, and have found the cheapest beans at Trader Joe’s work great. This is the Trader’s Joes Joe coffee, 14 ounces, approximately $5. I grind it in the store, coarsest grind. Leave the mess at the store .

This recipe makes 1 gallon. I make it in two half-gallon 64oz mason jars, with a Tube+ reusable filter in each.

Using a pair of pliers , crimp the edge of the coffee container at North-South-East-West

The coffee container, showing the 4 crimps.

Use your hand and pinch one side to its opposite. This will cause it to collapse, breaking on the crimps. Deforming the container makes it much easier to pour from.

Pour coffee in the filter, filling about half way.

Add two heaping tablespoons of Cacao Nibs to each filter. I use Terrasoul Cacao Nibs.

Add one heaping tablespoon of granulated chicory root. I use Starwest Botanicals Chicory Root Roasted Granules.

Add cold filtered water , dousing the grounds.

Stir with a kitchen butter knife , or other non-sharp utensil as you pour the water, trying to get all the grounds wet.

Add the remaining coffee grounds evenly to the two filters. Top off with water, stirring as you go.

Let the grounds sit, exposed, for about 30 minutes until all the gasses release. Then top off the grounds with water, filling to just under the top.

Seal with standard mason jar lid. Let stand at room temperature for 24-hours.

When brew is complete, open the lid, pull the filters up and hold for a few minutes while the brew drains .

Let the filter stand in a bowl for a few minutes to catch the remaining draining brew.

Pour the brew from the bowl back into the mason jars. The jars will now be approximately ⅔ full with brew. Fill the remainder of the jars with filtered water.

Chill in the refrigerator and drink. This photo shows the 1 gallon in the half-gallon mason jars, as well as a one-gallon mason jar with spout.

The recipe makes cold-brew concentrate. To drink, mix cold-brew concentrate 1:1 with water.

In my case I fill a glass ¼ with water, ¼ with half-and-half (light cream), and ½ with cold-brew concentrate. Top with a few ice cubes and ready to drink. In the winter I make a similar mix, but microwave for 3 minutes to get hot coffee.

Each gallon is approximately 20 cups of coffee . Cost is less than $0.50 per 12 oz cup. Very affordable.

Experiment by adjusting brew time, add-ins, and coffee beans. I find the Medium-Roast, Columbian, type beans work great. In the winter, some of the bolder Winter bean mixes are also good.

Writing Good git Commit Messages

Wed, 02 Jan 2019 09:00:00 -0500

Matej Jellus, writes :

A properly formed Git commit subject line should always be able to complete the following sentence: if applied, this commit will your subject line here. For example :

if applied, this commit will Delete unnecessary files

if applied, this commit will Add grep option

if applied, this commit will Fix error when protocol is missing

See the original post from Chris Beams

Mac Mouse Pointer Size - Change Your Life

Mon, 17 Dec 2018 09:00:00 -0500

This will change your life

I have recently setup a few new Mac laptops and am reminded how absurdly small the default mouse-pointer is on the Mac.

How to fix this:

Mac, menu, top-left of your screen
System Preferences…
Accessibility
Display
Cursor Size, adjust the slider, I prefer position 3
- I also recommend Reduce Motion and Reduce Transparency

iOS Cursor Trackpad Mode - Change Your Life

Thu, 13 Dec 2018 16:00:00 -0500

MacStories via John Gruber:

trackpad mode can be activated in a much easier way in iOS 12: just tap & hold on the space bar until the keyboard becomes a trackpad

This will change your life

On your iPhone and iPad, when you have the on-screen keyboard open and you are entering text, and you want to move around, usually to go back and fix a typo…

Put your finger on the space bar and hold
You will see the keyboard dim
Your finger now controls a trackpad
Move your finger around the keyboard and the cursor position on the screen follows
Release your finger, and you are back in keyboard mode with your cursor where you left it, ready to fix that typo

Update Sept 2019: For iPhone 11 and other newer devices without 3D Touch the process has changed a bit → Hold down the on-screen keyboard spacebar

iPad Web Development Tools

Wed, 14 Nov 2018 09:00:00 -0500

Craig Morey, writing via MacStories:

Could I recommend using an iPad as a good platform for doing front-end web development?

No.

Great analysis of tools, problems, and (partial) solutions for doing front-end web development on an iPad. It all applies to other forms of development in similar ways.

We aren’t there yet, but are getting closer.

Completely agree that Working Copy and Textastic are fantastic tools and get us closest to where we need to be.

As Frederico points out:

Make sure to watch the videos in Morey’s piece – I love how he detailed every single step of the workflows he tried to build on his iPad Pro

Arq Backup Monitor using AWS Lambda, API Gateway, S3

Tue, 13 Nov 2018 14:00:00 -0500

Arq Backup is a great personal backup product that lets you backup your Mac to your cloud storage provider of choice . I use Arq Backup for all the users of the many computers we have in the house. It works great, is fairly cheap, and gives you a high level of control over

what gets backed up (folder selection, exclusion rules)
when it gets backed up (scheduling, pause during hours)
security of backups (client side encryption)

One challenge is to make sure backups are running successfully for each user on each computer. If a backup doesn’t happen for 30 days I want to know about it.

I have constructed a two-part solution, Backup Marker and Lambda Monitor

Backup Marker

Write a marker meta-data file to AWS S3 after completion of each backup.

Create a new bucket to hold marker files arq-example-monitor
Create a service role for API Gateway that allows s3:PutObject into this bucket
Create an API Gateway service that integrates to S3 to upload the file
- Use service role created above
- Create an API Key, Usage Plan
- Set the API Gateway method to require the API Key
- Optionally set custom domain for the API
- Deploy the API
Add the API Key to your Mac keychain
- Replace zzz with API Key
- /usr/bin/security add-generic-password -a arq-finish-api-key -s arq-finish-api-key -w zzz -U -T /usr/bin/security
Create shell script arq-finish.sh to upload the marker file to S3 via API Gateway
- See code below, set the base URL of the API
- Put somewhere like /Users/Shared
- Give world execute, chmod 755 /Users/Shared/arq-finish.sh
Tell Arq to execute the shell script after each backup completes
- Set Arq -> Preferences -> Destinations -> (Select) -> Edit -> Before and After Backup -> Run shell script after backup completes to /Users/Shared/arq-finish.sh
- If one user has multiple backup destinations pass an additional argument to the script /Users/Shared/arq-finish.sh photo

arq-finish.sh

#!/bin/bash
BASE_URL=https://api.example.com/arq
API_KEY=`/usr/bin/security find-generic-password -a arq-finish-api-key -w`
ARG=${1:-main}
BACKUP_SET="`/bin/hostname -s`.`/usr/bin/whoami`.${ARG}"
FILE=${BACKUP_SET}.txt
URL=${BASE_URL}/${FILE}

/usr/bin/curl --header "x-api-key: $API_KEY" --upload-file - $URL << EOF
`/bin/date`
Arq `/usr/bin/defaults read /Applications/Arq.app/Contents/Info CFBundleVersion`
`/usr/bin/uname -a`  
EOF

Lamba Monitor

Periodically run a lambda function to check the age of the marker files and send an alert if too old.

Create SNS Topic ArqBackupMonitorAlert and subscribe your email address
Create a service role for Lambda which contains permission to s3:ListBucket and s3:GetObject on arq-example-monitor, and sns:Publish on ArqBackupMonitorAlert
Create ArqBackupMonitor Lambda
- Python 3 runtime
- Code below
- Set variables for at least ARQ_BUCKET and ARQ_SNS_ARN
- Create CloudWatch trigger to execute lambda every 10 days (set temporarily to every 5 minutes so you can test)

ArqBackupMonitor.py

import boto3
import os
from datetime import datetime, timezone, timedelta

def lambda_handler(event, context):
  arq_region = os.environ.get('AWS_REGION', 'us-east-1')
  arq_max_age_days = int(os.environ.get('ARQ_MAX_AGE_DAYS', '30'))
  arq_alert_subject = os.environ.get('ARQ_SNS_SUBJECT', 'Arq Monitor Alert')
  arq_bucket = os.environ.get('ARQ_BUCKET', 'arq-example-monitor')
  arq_alert_sns_arn = os.environ.get('ARQ_SNS_ARN', 
    'arn:aws:sns:us-east-1:12121212:ArqBackupMonitorAlert')
  
  too_old=''
  oldest_allowed = datetime.now(timezone.utc) - timedelta(days=arq_max_age_days)
  
  session = boto3.Session(region_name=arq_region)
  s3 = session.resource('s3')
  bucket = s3.Bucket(arq_bucket)
  for object in bucket.objects.all():
    if object.key.endswith('.txt'):
      if object.last_modified < oldest_allowed:
        too_old = too_old + object.key[:-4] + ' ' 
          + object.last_modified.strftime('%Y-%m-%d') + '\n'
 
  if len(too_old) > 0:
    msg = 'Arq backups are too old:\n\n' + too_old
    print(msg)
    sns = session.resource('sns')
    topic = sns.Topic(arq_alert_sns_arn)
    topic.publish(Subject=arq_alert_subject, Message=msg)
  
  return 'ArqBackupMonitor.Success'

Summary

After every successful Arq backup upload marker file via API Gateway to AWS S3
Every 10 days run a Lamba function that reads the marker files
If any marker file is older than 30 days send a notification to SNS Topic
SNS Topic sends notifcation via email

Note: Arq has a new product Arq Cloud Backup - full cloud backup solution including storage, like BackBlaze. I have installed Arq Cloud Backup up on my parents’ computers → it works great , “set it and forget it”. It also is incredibly fast, the Wasabi storage backend is speedy .

The Illustrated Network - Walter Goralski

Sun, 04 Nov 2018 17:00:00 -0400

After purchasing the 1st Edition hardcover of The Illustrated Network: How TCP/IP Works in a Modern Network by Walter Goralski and literally having it sit in my closet for 9 years, I finally decided to learn how IP addressing, networks, layers, routing… work. I donated the 1st edition and bought the revised 2nd epub edition as I wanted to have an updated and (literally) lightweight copy to keep on hand.

I always wondered how traceroute worked:

Whatever the type of traceroute packet, the time-to-live TTL field is initially set to 1, so the first router along the path should generate an ICMP Time Exceeded message to the sender. The round-trip delay in the timestamp field and IP address of the router is recorded by the sender and another set of packets is sent, this time with the TTL set to 2. These packets are discarded by the second router, and another ICMP message is sent back. The process is repeated until the destination host is reached and the host returns a Destination Port Unreachable message, or until a firewall is encountered that blocks the ICMP messages or unsolicited UDP traffic.

The end result should be a list of the routers on the path from source to destination that also records round-trip delays. In some cases (sometimes many cases), some routers will not respond to the TTL “timeout” with an ICMP message, but simply silently discard the offending packet. If the packet does not return within the timeout window, implementations indicate this with an asterisk (*) or some other placeholder and just keep going, trying to reach the next router.

It is a long book, but entirely worth it. Read the first 3/4 slowly and in detail taking notes along the way. The last 1/4 was more familiar already, so skimmed through most of it.

I recommend making a large printout out the example network diagram. It is refrenced throughout the book and helpful to have by your side.

Bottom Line: Highly recommended for anyone building or designing software systems (they are all networked, so adding that adjective is redundant). Especially those of us designing cloud architectures. Want to build proper VPCs, Security Groups, Placement Groups, Target Groups, Internet Gateways, Elastic IPs…? Best understand the fundamental concepts of networking. This book does the trick.

The supporting files for the 2nd Edition are not available, but after talking to Walter he told me they are the same as the 1st Edition supporting files.

Lambda CloudWatch Log Retain Manager

Wed, 31 Oct 2018 12:00:00 -0400

As mentioned in Lambda@Edge CloudFront Debugging, it is very common to have services, applications, and worldwide CloudFront Edge Locations (especially Lambda@Edge) creating CloudWatch Log Groups in regions across the world. By default new CloudWatch Log Groups have retention set to Never, which is never what I want.

I created a Lambda function that will search through all AWS regions, check the retention period of each CloudWatch Log Group. Any Log Group that has retention of Never will be changed to retention of 30 Days (configurable). I run this Lambda every 10 days via a scheduled CloudWatch rule. So far so good, keeping CloudWatch logs nice and tidy.

import boto3
import os

def lambda_handler(event, context):
  default_region = os.environ.get('AWS_REGION', 'us-east-1')
  retain_days = int(os.environ.get('RETAIN_DAYS', '30'))
  
  session = boto3.Session()
  client = session.client('ec2', region_name=default_region)

  for region_dict in client.describe_regions()['Regions']:
    region = region_dict['RegionName']
    print('Region:', region)
    logs = session.client('logs', region_name=region)
    log_groups = logs.describe_log_groups()

    for log_group in log_groups['logGroups']:
        log_group_name = log_group['logGroupName']
        if 'retentionInDays' in log_group:
          print(region, log_group_name, log_group['retentionInDays'], 'days')        
        else:
          print(region, log_group_name, retain_days, 'days **PUT**')        
          response = logs.put_retention_policy(
            logGroupName=log_group_name,
            retentionInDays=retain_days
          )

  return 'CloudWatchLogRetention.Success'

Although not used in this function, here is an example of the event object passed to the function when triggered by the CloudWatch scheduled rule

{
    "version": "0",
    "id": "arar40ba-aaaa-bbbb-cccc-81d33314b7",
    "detail-type": "Scheduled Event",
    "source": "aws.events",
    "account": "1212121212",
    "time": "2018-10-30T10:22:22Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:events:us-east-1:1212121212:rule/logrtnmgr"
    ],
    "detail": {}
}

Font Awesome Hugo Shortcode

Sun, 21 Oct 2018 13:30:59 -0400

Hugo Shortcodes

are simple snippets inside your content files calling built-in or custom templates

You can create custom Shortcodes to avoid adding raw html to your content. Below is a Shortcode to use Font Awesome icons without directly writing html.

Add the following to new file layouts/shortcodes/awesome.html

{{ if .Get 0}}
  <i class="{{ range .Params }}{{ . }} {{ end }}"></i>
{{ else }}
  <span class="shortcode-error">Shortcode usage: &lbrace;&lbrace;&lt; awesome 
  "far fa-smile-wink" &gt;&rbrace;&rbrace;</span>
{{ end }}

That code will add all the Shortcode parameters inside the class attribute of the i html tag. If there are no paramters the Shortcode will print out usage information to remind you how to use it.

Then in your Markdown content you can do something like:

here is a wink {{< awesome far fa-smile-wink >}}

Which generates html:

here is a wink <i class="far fa-smile-wink "></i>

And renders in the browser:

here is a wink

My Hugo-Coder Hugo Theme already has Font Awesome installed, but if yours doesn’t you can use the .HasShortcode function in your page template to see if that page is using the Font Awesome Shortcode and if so to include the Font Awesome Stylesheet in your page header.

Lambda@Edge CloudFront Debugging

Tue, 16 Oct 2018 17:30:00 -0400

James Beswick, writes on Medium:

but there’s a kink in The Edge process that has caught me out a few times. When your code runs at The Edge, the logging happens in the CloudWatch region of that edge server. The logs might not be where you think they will be.

Lesson: if you can’t find your logs and there are no 502 or 503 errors, The Edge is almost certainly logging in another region. It’s definitely logging somewhere.

Useful info on Lambda@Edge and the capturing of event samples and logging. Be careful when dealing with Lambda@Edge CloudFront CloudWatch logs. They won’t always show up in the region where you expect them to be. Very useful to know as you start to build and debug Lambda@Edge functions.

Here is a simple Lambda@Edge function that you can attach to any of the CloudFront events. It logs the CloudFront event and context passed to your Lambda.

exports.handler = (event, context, callback) => {
  const eventType = event.Records[0].cf.config.eventType;
  const responseEvent = eventType.indexOf('-response') > 0;
  
  console.log('DebugEvent ' + eventType 
    + ' -- ' + JSON.stringify(event)
    + ' -- context -- ' + JSON.stringify(context)
  );
  
  callback(null, (responseEvent ? 
      event.Records[0].cf.response : event.Records[0].cf.request));
};

Samples of the CloudFront context (which seems to be the same for all event types), and event objects for each event type (viewer-request, origin-request, origin-response, viewer-response) are packaged in a Github Gist

As James mentions, and you will find through experience, the Lambda@Edge CloudWatch logs are in the region where the CloudFront endpoint the user hit is located. So you will end up with CloudWatch logs all over the world. And CloudWatch by default sets no expiration on logs, which is troublesome. Below is a command-line script that will find and update log retention where it is unset. See the full gist for the code and the required IAM policy.

Update below log retain shell script converted into a proper lambda function

#!/bin/bash
AWS="/usr/local/bin/aws --profile log-retain-mgr --no-paginate --output json"
JQ="/usr/local/bin/jq --raw-output"

for REGION in `$AWS --region us-east-1 ec2 describe-regions | $JQ '.Regions[].RegionName'`
do
  echo "Region $REGION"
  for GROUP in `$AWS --region "$REGION" logs describe-log-groups | $JQ '.logGroups[] | select (has("retentionInDays") | not).logGroupName'`
  do
    echo "  $REGION $GROUP"
    $AWS --region "$REGION" logs put-retention-policy --log-group-name "$GROUP" --retention-in-days 30
  done
done

Text Notes

Fri, 12 Oct 2018 13:00:00 -0400

I take a lot of notes for work. Mainly notes about meetings:

Date
Location (if not a phone call, where was I, helps the memory)
Key Participants
Major Discussion points
Significant Decisions

I don’t do anything fancy. No attachments, no embedded content, no tagging. Just straight-up text. A touch of Markdown, but rarely rendered Markdown, just bullets and headers in the text for organization.

My notes and my email archive are, in effect, my work “memory”. MailMate on the Mac for all email, fantastic search, but that is a topic for another post.

I need to be able to get to my notes, easily, on multiple platforms: Mac OS, iPad, iPhone. This involves finding existing notes (search), and adding new note content.

Like David Sparks I have tried a number of note systems over the years, including Bear and Apple Notes. Anything that doesn’t store notes directly on the filesystem in text files doesn’t last long with me. Bear was great, but I didn’t like the files inside a SQLite database approach - having to run an export to have a backup was too much trouble. Besides, I have a dislike of app subscriptions, I can only subscribe to so many things, every month I am reminded to reconsider my decision. I would much rather a larger up-front purchase price, impulsive purchases are no-problem for me (a separate, but at least one-time, problem no doubt).

I was a long-time user of Notational Velocity and later Brett Terpstra’s nvALT on the Mac with 1Writer on iOS - all text file notes stored in 1Writer’s iCloud directory, nvALT pointed to that directory in Mobile Documents/ on Mac. It worked well, but development on nvALT long-ago ceased. Brett has said he is working on a successor to nvALT, which I eagerly await.

As of now my tools are

FSNotes $3 on iOS
Note files stored in FSNotes iCloud directory (sync)
The Archive $20 on the Mac, pointed at FSNotes’ Mobile Documents/ as the notes directory

The Archive has great UI, themes, and full-text search. So I can quickly find notes. FSNotes works well on iOS, able to find notes, and add content to notes from my iPad. FSNotes also has a Mac app, which works fine, and which I purchased to throw another $3 the developer’s way, but don’t use much.

My notes and text files are like weather apps, I just can’t stop fiddling with new apps, new systems, new approaches. So as expected, in the course of writing this post, I stumbled across Falcon. It seems to have all I want? Looks like another day will be spent fiddling around with my notes system

Update Looks like Falcon hasn’t been updated in over a year, I’ll pass

AWS Disaster Recovery

Thu, 11 Oct 2018 12:23:04 -0400

Corey Quinn, writing on Last Week in AWS:

if us-west-2 has an issue, and your plan is to fail over to us-east-1, you should probably be aware that more or less everyone has similar plans. You’ll be in very good, very plentiful company

expect provisioning times to be delayed significantly

Good point from Corey on this aspect of AWS disaster recovery planning. Of course you aren’t the only one with the same problem at the same time. If us-east-1 goes down, you are one of many many trying to failover and spin up new resources in us-west-2.

Basically a Thundering Herd style problem. So I guess the solution is

Know you are part of a herd
Beat the herd (idle, warm resources)
Don’t follow the herd (less popular region)