We started working this summer on laravel-data 4 together with typescript-transformer 3 (spoiler: that's my next priority, laravel-data 4 took longer than expected). After 136 commits, 316 files changed, 9306 lines removed, and a whopping 18143 lines added, it is finally out. Let's take a look at what's new!

A DataCollection-less world

When you had a collection of data objects nested in a data object, you were always required to use either a DataCollection, PaginatedDataCollection, or CursorPaginatedDataCollection to make sure the package would handle these objects as data objects. There was also a requirement to use the DataCollectionOf attribute to tell what was inside the collection.

These extra collections made code less readable; sometimes you want to use an array, not a whole collection. The support for extra methods like filter, reduce, ... was a much requested feature but it always felt that laravel-data was not the package that should implement those methods. Also, static analyzers and IDEs weren't always happy with these collections.

From now on, you're free to use array's, Laravel Collections, and paginators with data objects; the requirement to type what data object it stores is still there. But, an annotation can now be used, which will let your IDE provide much better completions than before:

class AlbumData extends Data
{
    public function __construct(
        public string $title,
        #[DataCollectionOf(SongData::class)]
        public DataCollection $songs,
    ){
    }
}

class AlbumData extends Data
{
    public function __construct(
        public string $title,
        /** @var SongData[] */
        public array $songs,
    ){
    }
}

The removal of the collection method

When creating a collection of data objects, you used the collection method in the past. That method has been removed in favor of the collect method to follow the Laravel convention.

Another reason for a new name is the changed behavior of the collect method concerning collection. The collect method will return the same collection type passed in. So, when passing in an array, you'll get an array with data objects returned. The collect method always returned either a DataCollection, PaginatedDataCollection, or CursorPaginatedDataCollection.

SongData::collection($anArray); // DataCollection<SongData>

SongData::collect($anArray); // SongData[]

If you still require a DataCollection, no problem! You can tell the collect method what to return:

SongData::collect($anArray, DataCollection::class); // DataCollection<SongData>

Magic collect

We already had magic methods for creating data objects using from. Now, we're extending this to collections, yet another reason to remove the collection method in favor of the collect method.

class SongData extends Data
{
    public function __construct(
        public string $name,
        public int $duration
    )
    {}
    
    public function collectLaravelCollection(Collection $collection): SongCollection
    {
        return new SongCollection($collection->all())
    }
}

Including, the included, includes

Laravel-data already had includes, which allowed you to construct an array from your data object how you liked it to be.

For example, we can make the songs property of AlbumData lazy:

class AlbumData extends Data
{
    public function __construct(
        public string $title,
        /** @var Lazy|SongData[] */
        public Lazy|array $songs,
    ){
    }
}

Transforming an album to an array will not include the songs property. You can add it to the array by manually including it:

$album->include('songs')->toArray();

The whole partials system, which ensures that includes, excludes, only, and except are performed correctly, was rewritten entirely in v3. We used a complex tree-based structure over there, which was complicated to work on, not performant at all, and stopped us from creating new features. My computer science background took over on that system, while the more pragmatic solution was far better.

The partials system is constantly being used when transforming data objects or collections into arrays and responses, ... it needs to be fast to have a performant application. That's why we've rewritten the whole partials system again! This time, it's structured much more straightforward, and we've ensured it is performant!

Next to performance, we tried to remove some kinks from the partial system. Previously, when adding a partial to a data object, that partial would sit there forever. The next time you transform an object, the partial will be executed again. In laravel-data 4, this isn't the case anymore:

$album->include('songs')->toArray(); // songs is included 
$album->toArray(): // songs is included

$album->include('songs')->toArray(); // songs is included 
$album->toArray(): // songs is NOT included

If you need partials that are always included, then add them to the includedProperties method on your data object or use the new includePermanently method:

$album->includePermanently('songs')->toArray(); // songs is included
$album->toArray(): // songs is included

Another change to the partials system is allowing inclusions to be defined wherever you want within your chain of data objects.

If we introduce another object:

class ArtistData extends Data
{
    public function __construct()
    {
        public string $name,
        /** @var Lazy|AlbumData[] */
        public Lazy|array $albums,
    }
}

Let's create the objects by getting them from the database:

$albums = Album::with('songs')->get()->map(function(Album $album){
    $albumData = AlbumData::from($album);
    
    if($albumData->title === 'Whenever You Need Somebody')
    {
        $albumData->include('songs');
    }
    
    return $albumData;
});

$artist = ArtistData::from([
    'name' => 'Rick Astley',
    'albums' => Lazy::create($albums)
]);

$artist->include('albums')->toArray();

As you can see, we include the albums on the root object, then add an include for the songs for one specific album.

Sidenote: for everyone who read the docs and tests of laravel-data, you know why I want to see the songs of this album ;)

In the previous versions, only the albums would be included, but not the songs of that specific album. You were always required to add includes only on the root level when transforming a data object into an array.

Laravel-data and its new partials system finally allow you to add includes wherever you want, meaning that the songs will be transformed into an array for that specific album.

New abstract classes

You must always extend from Data when creating data objects using the package. This class packs a lot of features like validation, transforming to arrays, casting, mapping property names, additional properties, and wrapping data.

In some situations, you might want to use leaner objects; that's why we've added Dto and Resource.

The Dto class allows you to create DTOs and nothing more; you can't transform them, they can't be wrapped, and they have no concept of includes making them ideal for simple objects to be used within your application.

As for the Resource objects, these mirror the Laravel resources; they allow data objects to be transformed into responses with wrapping, including properties and a lot more. But we've stripped the validation logic from them.

Creating data objects by factory

Each data object now has a new method, factory which allows you to create data objects using a factory pattern and define how the data object will be built.

It is possible, for example, to disable the mapping of property names and add some additional global casts:

SongData::factory()
    ->withoutPropertyNameMapping()
    ->withCast('string', StringToUpperCast::class)
    ->from($song);

In the factory, you can also configure whether magical creation is enabled wha, what magical methods to ignore, and whether payloads should be validated.

This factory creates a CreationContext object, which is then passed to all data, the cats, pipelines, and data objects being constructed.

Transforming data by factory pattern

The transformation process could not stay behind since the creation process can now be controlled with a factory.

The transformation method was changed to allow passing in a factory. You can now add extra partials, turn off transformation of values, turn off property name mapping, add wrapping, and a lot more:

$artistData->transform(
    TransformationFactory::create()
        ->incude('albums.songs')
        ->withWrapping()
        ->withTransformer('string', StringToUpperTransformer::class)
);

Kinda like the creation process, this factory will produce a TransformationContext, which will be passed to the transformers when transforming the data object to an array.

Validation strategies

A much-discussed subject on GitHub was whether all payloads provided to laravel-data should be validated. The default behavior is that requests always will be validated. The validateAndCreate method could be used if you want to validate something else.

In some situations, you want to be able to validate all the payloads passed to laravel-data. That's why you can now change the validation strategy to only requests, always, and disabled.

You can find this setting in the data.php config file or can use a factory for this:

SongData::factory()
    ->alwaysValidate()
    ->from($song);

Speed

We've added some benchmarks using phpbench to check how fast some operations are. If we compare the results of laravel-data v3 and v4, then we can see spectacular results:

benchCollectionTransformation...........I4 - Mo673.524μs (±1.57%)
benchObjectTransformation...............I4 - Mo49.853μs (±0.34%)
benchCollectionCreation.................I4 - Mo2.231ms (±0.75%)
benchObjectCreation.....................I4 - Mo149.323μs (±0.71%)

benchCollectionTransformation...........I4 ✔ Mo596.342μs (±1.23%)
benchObjectTransformation...............I4 ✔ Mo39.843μs (±0.84%)
benchCollectionCreation.................I4 ✔ Mo1.337ms (±0.56%)
benchObjectCreation.....................I4 ✔ Mo91.171μs (±0.81%)

As you can see, the transformation process is now a bit faster (10-15%), but the creation process is 40% faster!

A quick tip: these benchmarks were run with the structure caching feature we introduced in v3. Without this caching, the results look like this:

benchCollectionTransformationWithoutCac.I4 ✔ Mo797.039μs (±2.93%)
benchObjectTransformationWithoutCache...I4 ✔ Mo233.757μs (±1.72%)
benchCollectionCreationWithoutCache.....I4 ✔ Mo1.627ms (±1.45%)
benchObjectCreationWithoutCache.........I4 ✔ Mo351.498μs (±0.72%)

Always cache your structures when deploying to production!

A revamped type system

Laravel-data packs a lot of magic to create and transform objects. Therefore, it needs a lot of PHP reflection and other tricks to make it work.

The type system is one of those tricks; it keeps track of the types within your data objects. For laravel-data v4, this system was entirely built up from the ground again (twice actually; the first implementation was trashed after a few months) to be more versatile and performant.

The most notable change you'll probably see of this rewrite is better performance and support for PHP 8.2 BNF types:

class BnfData extends Data
{
    public function __construct(
        public (InterfaceA&InterfaceB)|null $aBNFType
    )
}

Small changes

• We now require Laravel 10. As soon as Laravel 11 comes out, we'll, of course, support it
• Laravel Model attributes can now also be used to create data
• The from method now can be called without arguments
• The docs have been rewritten to be more coherent
• We've restructured the tests around the different features of the package to make testing easier

For this release we've also created an extensive upgrade guide, you'll find it here.

What's next

I've been working hours on this release for the last weeks, neglecting new PRs and issues. Next week, I'm going to skim through them all. Plus, I expect a lot of new issues and PRs for this new version of the package.

After that, it is time to work further on typescript-transformer v3, which is completely rewritten but needs some more love.

After typescript-transformer, I'm going to start looking at laravel-data 5. This will be a validation-heavy release where we try to iron out the kinks in the validation process as we did in this release with the partials system.

The web is an open place with many bad actors, so a signature is sent with this request as a protection measure. This signature is constructed from a secret key that both sender and receiver know and the contents of the payload.

In PHP, the sender would create a request and then a signature as follows:

$signature = hash_hmac(
 'sha256',
 $payload,
 'some_secret_key'
);

This signature is then appended as a header to the request.

The receiver on the other side would then again construct the signature like above and then check if it corresponds with the signature sent:

hash_equals($request->header('signature'), $signature);

If the two correspond, we know no one tampered with the message, and the request was sent by the service we trust (not some other bad actor who found out what the endpoint was).

Local signature mess

That's all great. Until you start debugging errors why something is not working, Postman is one of the best tools to send API requests from your computer when you're debugging.

But since the signature depends on the contents of the request payload, it needs to be recalculated each time the payload changes.

Luckily Postman has a solution for this! It is possible to run some JavaScript code before the request is sent, which allows you to change the request right before it is sent out. You can write this code within the "Pre-request Script" tab:

To add a signature header, we use this piece of code:

const secret ='some_secret_key';

var signature = CryptoJS.HmacSHA256(request.data, secret).toString(CryptoJS.digest);

pm.request.addHeader({
 key: 'signature',
 value: signature
});

console.log("Signature :: " + signature)

This code will add a signature header with an HMAC SHA256 hashed version of the secret and payload content, neat!

When debugging errors, we as developers have a few tools available: an exception of a particular class, a message, a status code, and a stack trace. The stack trace is a report that provides information about the sequence of function calls that led to an error or an exception. It shows the order in which functions were called.

We name each call to a function/method within a stack trace a frame. It has lots of useful info like the file name, line number, function name, class (sometimes), and arguments that were used to make the call. Having arguments within a stack trace can be dangerous. For example, let's say you have a function like this:

function getPayments(string $apiKey): array 
{
    throw new Exception('Could not fetch payments');
}

Usually, you would have some application code here, fetching payments through an external API, but to simplify things, let's go straight to the case where we throw an exception because the payments couldn't be fetched.

You can call this function as such:

getPayments('dj8gd4sl05db32xjch7989dsxws');

This function call, of course, fails miserably, providing you with the following exception message:

Fatal error: Uncaught Exception: Could not fetch payments in /in/UoFa1:6
Stack trace:
#0 /in/UoFa1(32): getPayments('dj8gd4sl05db32x...')
#1 {main}
  thrown in /in/UoFa1 on line 6
<br/><i>Process exited with code <b title="Generic Error">255</b>.</i>

Take a look at the third line:

#0 /index.php(34): getPayments('dj8gd4sl05db32x...')

That's our API key in plain sight, which we want to avoid at all costs!

Above, we saw the stringified version of the stack trace. When we catch the exception, we can take a look at a more normalized array version:

try {
    getPayments('dj8gd4sl05db32xjch7989dsxws');
} catch (Exception $e) {
    var_dump($e->getTrace());
}

Catching the exception gives us the following stack trace:

array(1) {
  [0]=>
  array(4) {
    ["file"]=>
    string(9) "/index.php"
    ["line"]=>
    int(34)
    ["function"]=>
    string(11) "getPayments"
    ["args"]=>
    array(1) {
      [0]=>
      string(27) "dj8gd4sl05db32xjch7989dsxws"
    }
  }
}

As you can see, we're passing one argument to the getPayments function, which can be helpful when debugging. But this information can also end up in the wrong hands when displayed in logs or error pages like Ignition.

That's the reason why we refrained from showing these parameters in Ignition:

And thus, due to the tight coupling of Flare and Ignition, these parameters won't end up in Flare.

But this all might change due to a recent addition to PHP 8.2.

Sensitive Parameters

Starting from PHP 8.2, some parameters can now be tagged as 'sensitive' using an attribute, meaning a parameter will be hidden from a stack trace.

Let's rewrite our getPayments function as follows:

function getPayments(
    #[SensitiveParameter]
    string $apiKey
): array {
    throw new Exception('Could not fetch payments');
}

Now when we execute the function, we get the following:

Fatal error: Uncaught Exception: Could not fetch payments in /in/Ko3d8:13
Stack trace:
#0 /in/Ko3d8(32): getPayments(Object(SensitiveParameterValue))
#1 {main}
  thrown in /in/Ko3d8 on line 13
<br/><i>Process exited with code <b title="Generic Error">255</b>.</i>

The API key is replaced with Object(SensitiveParameterValue), hiding the original value! Let's see how the stack trace looks:

array(1) {
  [0]=>
  array(4) {
    ["file"]=>
    string(9) "/in/cv0er"
    ["line"]=>
    int(33)
    ["function"]=>
    string(11) "getPayments"
    ["args"]=>
    array(1) {
      [0]=>
      object(SensitiveParameterValue)#2 (0) {
      }
    }
  }
}

Again the API key is gone from traces, but the value still exists. The original string value is replaced with an object SensitiveParameterValue, which wraps the original value. We can see this by retrieving the original value as such:

try {
    getPayments('dj8gd4sl05db32xjch7989dsxws');
} catch (Exception $e) {
    var_dump($e->getTrace()[0]['args'][0]->getValue()); // dj8gd4sl05db32xj...
}

Classes

The example mentioned above, used functions to make things easy to understand, but you can perfectly tag parameters as sensitive in a class:

class PaymentApi
{
    public function getPayments(
    	 int $tenantId,
        #[SensitiveParameter]
        string $apiKey
    ): string {
        throw new Exception('Could not fetch payments');
    }
}

Running this:

try {
    $api = new PaymentApi();
    $api->getPayments(314, 'dj8gd4sl05db32xjch7989dsxws');
} catch (Exception $e) {
    var_dump($e->getTrace());
}

It gives us the following stack trace:

array(1) {
  [0]=>
  array(6) {
    ["file"]=>
    string(90) "/index.php"
    ["line"]=>
    int(36)
    ["function"]=>
    string(11) "getPayments"
    ["class"]=>
    string(10) "PaymentApi"
    ["type"]=>
    string(2) "->"
    ["args"]=>
    array(2) {
      [0]=>
      int(314)
      [1]=>
      object(SensitiveParameterValue)#3 (0) {
      }
    }
  }
}

As you can see, the tenantId is a non-sensitive parameter, so its value still appears in the stack trace. However, apiKey, the sensitive parameter value, has been wrapped into a SensitiveParameterValue object.

Flare

As mentioned, we don't support stack trace arguments in Ignition for security reasons, and this new sensitive parameter feature might let us rethink this. On our roadmap, I've created a new feature request, allowing arguments to be shown in Ignition and Flare stack traces. Give it a thumbs up if you want to see this implemented.

We're still working on our redesign and are close to release. Today we started tackling an issue where the performance of the error page was too slow for us. In the end, we needed to change the structure of the error occurrences table, which is a lot harder than it seems.

In the redesign of Flare, we've added a new feature called insights which gives you a quick overview of some statistics like which endpoints triggered the error, which users, which application versions, and so on.

The counts for these insights are always live calculated, which means queries like this:

SELECT
	count(DISTINCT entry_point) AS aggregate
FROM
	`error_occurrences`
WHERE
	`error_occurrences`.`error_id` = ?
	AND `error_occurrences`.`error_id` IS NOT NULL
	AND `entry_point_type` = 'web';

Each error occurrence has an entry_point_type (web, queue, command) and an entry_point (a URL, job class, or command). These queries were quite performant on our seeded data, but in production, we have an error that has accumulated 66k occurrences. That's the point where things start to become slow.

Luckily, there's an easy solution to this, indexes! So we've added an index to entry_point_type and saw a performance boost. Later on, we tried adding an index to entry_point, but this exception popped up:

BLOB/TEXT column 'entry_point' used in key specification without a key length

Wait! Are we using a text type column for entry_point? That's a bit crazy. These columns are stored differently than varchars in MySQL, which makes them a lot slower + indexing them has some limitations. Let's fix this using a varchar column, but how many characters should it be?

The first thing we checked was how many characters are required on average. A quick query that counted the number of occurrences for each entry_point length gave us the following:

We're pretty safe with 255 characters because almost all entry points for occurrences are around that length. Because a URL is somewhat limited to 2048 characters and varchars smartly assign space (it only takes the space required for a value), we took 2048 characters for this column.

Now changing our column can be done as such:

ALTER TABLE `flare`.`error_occurrences`
CHANGE `entry_point` `entry_point` varchar(2048) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NULL;

The thing is, this is a large table at the moment. There are 10531985 it.. 10532202 ite.. 10532278 items. You get the point. A lot of items, and the table is constantly growing.

By default, MySQL will make a copy of this table, apply the changes to that table, remove the old table, and use the newly created one instead. This process is extremely slow, and it blocks a lot of statements, completely shutting down Flare. We should avoid this at all costs.

The trick with the extra column

There are a few solutions to this problem. In our case, we did the following: we first created a new column called entry_point_2 with the correct type:

ALTER TABLE `flare`.`error_occurrences` 
ADD COLUMN `entry_point_2` varchar(2048) NULL, ALGORITHM=INSTANT;

The MySQL instant algorithm will instantly add the column (it still takes some time, but no locks are required). The problem? When we ran this query, MySQL started copying our table, which was the thing we were trying to avoid.

After some research work in the MySQL documentation, we discovered that these operations will still use the copy algorithm if a fulltext index exists on the table, even when that index isn't used in the operation.

So the easy fix was to drop the fulltext index temporarily:

ALTER TABLE `flare`.`error_occurrences` 
DROP INDEX `error_message_fulltext`;

Great, we can now add an extra column without too much hassle.

Moving data

Next, we move the data from entry_point to entry_point_2. This operation is going to take some time. The cool thing is we can already ensure that newly created occurrences immediately fill entry_point_2.

Doing such a thing would require some changes to the code, then deploying that code, later changing the code, deploying that code again ... too much complexity if you ask me.

MySQL has a feature called triggers, which are small hooks running before or after inserting, updating, or deleting a row. Since we only add error occurrences, an insert trigger was enough to make sure each new occurrence has its entry_point_2 set:

CREATE TRIGGER `flare`.`error_occurrences`
	BEFORE INSERT ON `error_occurrences`
	FOR EACH ROW
	BEGIN
	SET NEW.`entry_point_2` = NEW.`entry_point`;
END

Now we can update all the other entries within the table which are missing a value for entry_point_2:

UPDATE
	error_occurrences
SET
	entry_point_2 = entry_point
WHERE
	entry_point IS NOT NULL
	AND entry_point_2 IS NULL

There are probably more performant ways to this (it took a whopping 20 minutes to execute this query), but it doesn't add locks to our table and is easy to write.

Setting everything right

So we're almost there. We've got identical columns, entry_point of type text and entry_point_2 of type varchar(2048). We're going to rename these columns so that:

• entry_point -> old_entry_point
• entry_point_2 -> entry_point

We can do this instantly like this:

ALTER TABLE  `flare`.`error_occurrences` 
RENAME COLUMN entry_point TO old_entry_point, 
RENAME COLUMN entry_point_2 TO entry_point;

We remove the trigger we've created:

DROP TRIGGER `flare`.`error_occurrences`;

And in the end, altogether remove the old_entry_point column:

ALTER TABLE `flare`.`error_occurrences` 
DROP COLUMN `old_entry_point`, ALGORITHM=INSTANT;

Now we can create an index on the newly created entry_point column:

ALTER TABLE `flare`.`error_occurrences`
ADD INDEX `error_occurrences_entry_point_index` (`entry_point`(255)) USING BTREE;

And, of course, we also need to create the fulltext index we removed earlier. The thing is, we can only add such an index when nothing is written to the table. To fix this, we temporarily paused Laravel Horizon:

php artisan horizon:pause

Our queues are now paused so that nothing gets written to this table. We don't lose anything thanks to Horizon because the jobs we've added still exist, and they can be executed as soon as we restart Horizon again.

Now we can add the fulltext index as such:

ALTER TABLE `flare`.`error_occurrences`
ADD FULLTEXT INDEX `error_message_fulltext` (`exception_message`);

And restart horizon:

php artisan horizon:continue

Conclusion

A minor fix turned out to be more complicated than expected, but the result is what counts. And our page now got a lot faster. We'll start inviting beta testers soon. Interested? Contact us at support@flareapp.io.

Our whole team is currently working hard on the next version of Flare. We're completely redesigning the application and website, and it is an immense effort, but it will undoubtedly pay off in the future.

In this rewrite of Flare, we decided to move from Laravel resources to our own Laravel data package for sending data to the front end. Doing this not only adds the benefit of a fully typed response, but we also get TypeScript definitions for our resources without extra effort.

Once all resources were manually converted into data objects, we were excited to check if everything worked fine. Though we didn't hit on any exceptions or errors, some application pages felt extremely slow, too slow to be useful for our clients.

We quickly found that much of the response time was spent on the data objects. We started by optimizing the data objects using the lazy functionality, which allows loading certain parts of the data later on when the initial page is already sent. It made the pages faster, but we didn't want to stop there since we're completely redesigning the whole application.

The laravel-data package is fantastic to work with, but it also adds a lot of complexity when outputting data. In this blog post, we will look at how we've improved the performance of the package and, thus, the complete Flare application.

New tools to play with

To get started, I will introduce you to three tools you can use within your toolset to research performance problems: Xdebug, PHPBench, and QCacheGrind.

Xdebug is a PHP extension that provides debugging and profiling capabilities. It is most famous for its complicated configuration, which may slow down your whole application. Luckily, Xdebug version 3 made extensive progress on this part, and setting up Xdebug is now relatively easy.

PHPBench is a benchmarking tool for PHP that can help you measure the performance of your code. It allows you to create test cases and run them multiple times to get a notion of the speed of your code.

Lastly, QCacheGrind is an application allowing you to look at the profiles generated by Xdebug visually. QCacheGrind is available for MacOS and Windows. If you're on Linux, look at KCachegrind, which is almost the same.

Getting a baseline

To start, we need to be able to test if the changes we make to our code have a positive or negative impact on the performance. The best way to do this is by creating a few benchmarks using PHPBench. These benchmarks will be run multiple times, and the average time this take can be used as a metric to see if we've made a difference by updating the code.

PHPBench works a lot like PHPUnit. You create a few benchmark files (like tests) with benchmark cases (like test cases). The PHPBench command will then execute these files like the PHPUnit command would. Only it executes the cases multiple times and tracks how long it takes.

We get started as usual by installing PHPBench:

composer require phpbench/phpbench --dev

Next, we create a phpbench.json file which configures PHPbench:

{
     "$schema":"./vendor/phpbench/phpbench/phpbench.schema.json",
     "runner.bootstrap": "vendor/autoload.php",
     "runner.path" : "benchmarks"
 }

The runner.path option defines the directory where our benchmarks are located, so we create a new directory for them:

mkdir benchmarks

The last thing to do is create the benchmark. We call this file ExampleBench.php. Notice the Bench suffix. Using this suffix is essential because, otherwise, PHPBench won't execute the file. Within the ExampleBench.php, we add an ExampleBench class:

class ExampleBench
{
     
}

Let's create a first benchmark:

use PhpBench\Attributes\Iterations;
use PhpBench\Attributes\Revs;

class ExampleBench
{
    #[Revs(500), Iterations(5)]
    public function benchPow()
    {
		pow(2, 10);
    }
}

As you can see, we will benchmark how long it takes to calculate 2^10. The benchmark case we've added is called benchPow. Notice the bench keyword prefixed, which is required for PHPBench to find the benchmark case.

We've also added two attributes to the method:

Revs, short for revolutions, refers to the number of times the benchmark is executed consecutively within a single time measurement. The more revolutions, the more accurate the result. In this case, we will calculate 2^10 a whopping 500 times!

In an ideal world, we're done and can start measuring. But running these revolutions multiple times is safer and more correct to ensure our measurements are stable and don't differ too much.

We have five iterations of 500 revolutions so that the pow function will be executed for 5 * 500 = 2500 times.

Now it is time to run PHPBench:

vendor/bin/phpbench run  --report=default

And we get the following result:

PHPBench (1.2.9) running benchmarks... #standwithukraine
with configuration file: /Users/ruben/Spatie/laravel-data/phpbench.json
with PHP version 8.2.3, xdebug ✔, opcache ❌

\ExampleBench

    benchPow................................I4 - Mo0.038μs (±3.33%)

Subjects: 1, Assertions: 0, Failures: 0, Errors: 0
+------+--------------+----------+-----+------+------------+----------+--------------+----------------+
| iter | benchmark    | subject  | set | revs | mem_peak   | time_avg | comp_z_value | comp_deviation |
+------+--------------+----------+-----+------+------------+----------+--------------+----------------+
| 0    | ExampleBench | benchPow |     | 500  | 1,812,376b | 0.036μs  | -1.58σ       | -5.26%         |
| 1    | ExampleBench | benchPow |     | 500  | 1,812,376b | 0.038μs  | +0.00σ       | +0.00%         |
| 2    | ExampleBench | benchPow |     | 500  | 1,812,376b | 0.038μs  | +0.00σ       | +0.00%         |
| 3    | ExampleBench | benchPow |     | 500  | 1,812,376b | 0.038μs  | +0.00σ       | +0.00%         |
| 4    | ExampleBench | benchPow |     | 500  | 1,812,376b | 0.040μs  | +1.58σ       | +5.26%         |
+------+--------------+----------+-----+------+------------+----------+--------------+----------------+

It looks like 0.038μs is the best average we have. Let's see what's the fastest:

1. 2^20
2. 2^10 * 2^10

Our benchmark now will look like this:

class ExampleBench
{
    #[Revs(500), Iterations(5)]
    public function benchPow()
    {
        pow(2, 20);
    }

    #[Revs(500), Iterations(5)]
    public function benchPowPow()
    {
        pow(2, 10) * pow(2, 10);
    }
}

Running PHPBench gives the following results:

PHPBench (1.2.9) running benchmarks... #standwithukraine
with configuration file: /Users/ruben/Spatie/laravel-data/phpbench.json
with PHP version 8.2.3, xdebug ✔, opcache ❌

\ExampleBench

    benchPow................................I4 - Mo0.039μs (±3.78%)
    benchPowPow.............................I4 - Mo0.060μs (±11.25%)

Subjects: 2, Assertions: 0, Failures: 0, Errors: 0
+------+--------------+-------------+-----+------+------------+----------+--------------+----------------+
| iter | benchmark    | subject     | set | revs | mem_peak   | time_avg | comp_z_value | comp_deviation |
+------+--------------+-------------+-----+------+------------+----------+--------------+----------------+
| 0    | ExampleBench | benchPow    |     | 500  | 1,812,376b | 0.040μs  | +0.27σ       | +1.01%         |
| 1    | ExampleBench | benchPow    |     | 500  | 1,812,376b | 0.042μs  | +1.60σ       | +6.06%         |
| 2    | ExampleBench | benchPow    |     | 500  | 1,812,376b | 0.038μs  | -1.07σ       | -4.04%         |
| 3    | ExampleBench | benchPow    |     | 500  | 1,812,376b | 0.038μs  | -1.07σ       | -4.04%         |
| 4    | ExampleBench | benchPow    |     | 500  | 1,812,376b | 0.040μs  | +0.27σ       | +1.01%         |
| 0    | ExampleBench | benchPowPow |     | 500  | 1,812,376b | 0.076μs  | +1.47σ       | +16.56%        |
| 1    | ExampleBench | benchPowPow |     | 500  | 1,812,376b | 0.072μs  | +0.93σ       | +10.43%        |
| 2    | ExampleBench | benchPowPow |     | 500  | 1,812,376b | 0.060μs  | -0.71σ       | -7.98%         |
| 3    | ExampleBench | benchPowPow |     | 500  | 1,812,376b | 0.060μs  | -0.71σ       | -7.98%         |
| 4    | ExampleBench | benchPowPow |     | 500  | 1,812,376b | 0.058μs  | -0.98σ       | -11.04%        |
+------+--------------+-------------+-----+------+------------+----------+--------------+----------------+

So basically, it costs 50% more time to calculate two pow(2, 10) functions than one pow(2, 20), that's quite useful!

Now back to laravel-data, we want to know how fast we can create a data object and how fast we can transform it into a JSON object. We can describe this as follows:

class DataBench
{
	#[Revs(500), Iterations(2)]
    public function benchDataCreation()
    {
        MultiNestedData::from([
            'nested' => ['simple' => 'Hello'],
            'nestedCollection' => [
                ['simple' => 'Flare'],
                ['simple' => 'is'],
                ['simple' => 'awesome'],
            ],
        ]);
    }

    #[Revs(500), Iterations(2)]
    public function benchDataTransformation()
    {
        $data = new MultiNestedData(
            new NestedData(new SimpleData('Hello')),
            new DataCollection(NestedData::class, [
                new NestedData(new SimpleData('Flare')),
                new NestedData(new SimpleData('is')),
                new NestedData(new SimpleData('awesome')),
            ])
        );

        $data->toArray();
    }

    #[Revs(500), Iterations(2)]
    public function benchDataCollectionCreation()
    {
        $collection = Collection::times(
            15,
            fn() => [
                'nested' => ['simple' => 'Hello'],
                'nestedCollection' => [
                    ['simple' => 'Flare'],
                    ['simple' => 'is'],
                    ['simple' => 'awesome'],
                ],
            ]
        )->all();

        MultiNestedData::collection($collection);
    }

    #[Revs(500), Iterations(2)]
    public function benchDataCollectionTransformation()
    {
        $collection = Collection::times(
            15,
            fn() => new MultiNestedData(
                new NestedData(new SimpleData('Hello')),
                new DataCollection(NestedData::class, [
                    new NestedData(new SimpleData('Flare')),
                    new NestedData(new SimpleData('is')),
                    new NestedData(new SimpleData('awesome')),
                ])
            )
        )->all();

        $collection = MultiNestedData::collection($collection);

        $collection->toArray();
    }
}

In the first benchmark, we're creating a data object using arrays. In the second one, we created an object as efficiently as possible by manually defining it and then transforming it to JSON.

We also added two cases where we do the same, but instead of using data objects, we benchmark using data collections of multiple data objects.

Now running PHPBench will fail because the laravel-data package depends on some Laravel functionality. Luckily Laravel provides excellent testing infrastructure, which we can use within our benchmark. We do this by using the CreatesApplication trait, which is also present in the base Laravel test case:

use Tests\CreatesApplication;

class DataBench
{
    use CreatesApplication;

    public function __construct()
    {
        $this->createApplication();
    }
    
    // The benchmarks
}   

We need another update because we're benchmarking a laravel package, not a laravel application. This means we must use the CreatesApplication trait from the orchestra/testbench package, which is used to test Laravel packages. And we also need to specify the laravel-data service provider to boot up the package:

use Orchestra\Testbench\Concerns\CreatesApplication;

class DataBench
{
    use CreatesApplication;

    public function __construct()
    {
        $this->createApplication();
    }

    protected function getPackageProviders($app)
    {
        return [
            LaravelDataServiceProvider::class,
        ];
    }
    
    // The benchmarks
} 

Great, we now have some benchmarks ready to measure the speed of our code!

Getting started with Xdebug

Next up, we need to profile our code. When we're profiling our code, we will run it a bit differently than like would typically. When profiling the PHP process will write down each function call we make and also keep track of how long it takes to run the function. Ultimately, all this information will be written into a file we can analyze.

To get this working, we need to install and enable Xdebug. You can find the instructions here.

For me, on a Mac, it was as simple as this:

sudo pecl install xdebug

We also need to update or create the Xdebug .ini file:

[xdebug]

xdebug.mode=profile
xdebug.start_with_request=yes

On my Mac, I've put this file here: nano /opt/homebrew/etc/php/8.2/conf.d/xdebug.ini.

We configure two options:

• xdebug.mode=profile to enable profiling in Xdebug.
• xdebug.start_with_request=yes to start profiling with PHPBench. When you're done with profiling, don't forget to set this to off or trigger. Otherwise, all your PHP applications will take ages to load.

We have our benchmarks, and we have our profiler. Now we only need to combine those two, and then we can discover why our code is so slow!

Profiling the code

We will rerun our benchmarks, but this time we will also profile the benchmarks. We can do this with the following command:

vendor/bin/phpbench xdebug:profile

Which gives the following output:

PHPBench (1.2.9) running benchmarks... #standwithukraine
with configuration file: /Users/ruben/Spatie/laravel-data/phpbench.json
with PHP version 8.2.3, xdebug ✔, opcache ❌

\DataBench

    benchDataCreation.......................I0 - Mo10.762ms (±0.00%)
    benchDataTransformation.................I0 - Mo1.095ms (±0.00%)
    benchDataCollectionCreation.............I0 - Mo159.879ms (±0.00%)
    benchDataCollectionTransformation.......I0 - Mo13.952ms (±0.00%)

Subjects: 4, Assertions: 0, Failures: 0, Errors: 0

4 profile(s) generated:

/Users/ruben/Spatie/laravel-data/.phpbench/xdebug-profile/3fcb020036c8d7e8efdcddd4dbd66b92.cachegrind.gz
/Users/ruben/Spatie/laravel-data/.phpbench/xdebug-profile/8deba1dcce573e1bf818772ac7a5ace0.cachegrind.gz
/Users/ruben/Spatie/laravel-data/.phpbench/xdebug-profile/06d049dacb2a7a3809e069c6c8289e02.cachegrind.gz
/Users/ruben/Spatie/laravel-data/.phpbench/xdebug-profile/41c9fe618b93431786fff90b1d624b82.cachegrind.gz

Running the benchmark suite now takes way longer than before. In the end, we have a new directory with our profiles:

• .phpbench/xdebug-profile/3fcb020036c8d7e8efdcddd4dbd66b92.cachegrind.gz
• .phpbench/xdebug-profile/8deba1dcce573e1bf818772ac7a5ace0.cachegrind.gz
• .phpbench/xdebug-profile/06d049dacb2a7a3809e069c6c8289e02.cachegrind.gz
• .phpbench/xdebug-profile/41c9fe618b93431786fff90b1d624b82.cachegrind.gz

Each file corresponds to a benchmark case within your benchmark suite. The best way to find out which file belongs to which benchmark is to look at the execution order of benchmark cases and match them with the files in the order they were created.

At the moment, all files are compressed. Let us decompress them:

gzip --decompress .phpbench/xdebug-profile/*

Analyzing

It is finally time to find out why our code is slow. This is also the most complicated step in the process, and there is no guide to quickly finding a bottleneck. You should have a good understanding of your code and be able to see strange behavior within profiles. Some things that might stand out:

• functions that shouldn't be executed
• functions that are executed too much
• functions that take longer than expected

Let's open a profile where we create a collection of data objects:

First, we're going to configure QCacheGrind a bit more. Go to:

QCacheGrind -> Preferences -> Source Annotations

And add the path where your code is situated. This allows QCacheGrind to give some hints with code examples of which code is slow.

The application has three panels:

• On the left: the functions which were called sorted by the lengthiest call
• On the right-top: the callers of a selected function
• On the right-bottom: for a selected function, the functios called

On the left side, we see all the functions called. The Self metric shows how long it took to call a function, the Incl. metric shows how long the function and all the functions it called took. So the PHPBench function, for example, takes 99.93% of the time Incl. but it spends only 0.01% of the time within the function. All the other time is used to call other functions, our benchmarks with data creation.

Immediately we see something bizarre. We spend 24.34% of the time in Illuminate\Container\Container->resolve(). Let's click on that function and see what's happening:

This function has been called by Illuminate\Foundation\Application->resolve(). Let's click on that one. We see Illuminate\Container\Container->make() as the function, usually calling the resolve function. We keep on clicking on the most called functions until we enter the helpers.php file:

From this data, we can see that we're resolving a lot from the Laravel container:

• Data::from 67500 items
• DataPipeline in a closure (line 75) -> 225000 items
• DataPipeline in a closure (line 69) -> 187500 items

What's exactly happening here? Let's take a look at the DataPipeline code:

public function execute(): Collection
{
    /** @var \Spatie\LaravelData\Normalizers\Normalizer[] $normalizers */
    $normalizers = array_map(
        fn (string|Normalizer $normalizer) => is_string($normalizer) ? app($normalizer) : $normalizer,
        $this->normalizers
    );

    /** @var \Spatie\LaravelData\DataPipes\DataPipe[] $pipes */
    $pipes = array_map(
        fn (string|DataPipe $pipe) => is_string($pipe) ? app($pipe) : $pipe,
        $this->pipes
    );
    
    // Other code, hidden to keep things a bit easier to read
}

This code doesn't look like a performance problem. Yes, we're creating these normalizers and pipes from the container, but as long as this method isn't being executed that many times there won't be a problem in the end. This process should happen somewhere.

Let's find out from where this code is called:

public function execute(string $class, mixed ...$payloads): BaseData
{
    $properties = new Collection();

    foreach ($payloads as $payload) {
        /** @var BaseData $class */
        $pipeline = $class::pipeline();

        foreach ($class::normalizers() as $normalizer) {
            $pipeline->normalizer($normalizer);
        }

        foreach ($pipeline->using($payload)->execute() as $key => $value) {
            $properties[$key] = $value;
        }
    }
    
    // Other code, hidden to keep things a bit easier to read
}

That could be better. We create the pipeline for each data object we're constructing, and thus, we resolve all the pipes and normalizers from the container each time. Every time again, this is not required as the pipeline is statically constructed for a data object and thus the same for all objects.

For a few objects, this will be fine. But since we have collections of the same objects (sometimes more than 500), we hit a huge performance problem.

We now cache these pipelines in version 3.2 of laravel-data and add some other performance improvements. This should do a lot for our performance! Finally, let's rerun the benchmarks.

Before performance optimizations:

benchData...............................I1 - Mo293.376μs (±0.79%)
benchDataCollection.....................I1 - Mo5.619ms (±0.48%)

After performance optimizations:

benchData...............................I1 - Mo125.731μs (±0.57%)
benchDataCollection.....................I1 - Mo2.111ms (±0.82%)

For plain data objects, we've improved performance by 57%, and for collections of data, by 62%. That's huge!

In the end

With a few hours of work, we drastically reduced the performance penalty on Flare. Profiling the most critical parts of your application can be valuable, but it is not that easy!

We keep redesigning and refactoring Flare for the next weeks and hope to launch soon! You can find a preview of our new design here. Are you interested in beta testing the new version of Flare? Please send us an email at support@flareapp.io.

Since the early days, Laravel includes an excellent validator class. It allows you to check if the input provided to your application is valid according a set of rules and follows a specific format. Laravel's validator works on any arbitrary array of data but it's typically used for request validation. You can check if an email is valid, a password is confirmed, a file is a pdf, a string is not longer than a certain amount of characters, and so much more.

Validating input in Laravel is incredibly powerful and quick to set up. It's always been one of our favorite Laravel features. But, a few strange edge cases can be found where the validator isn't working as expected. When developing a Flare feature, we encountered one of these oddities and found a nice way to work around it.

Suppose that we're trying to validate a basic array with a nested team property:

[
	'name' => 'Ruben Van Assche',
	'team' => [
		'id' => 1,
		'role' => 'engineer'
	]
]

(The example above is abbreviated for better readability and clarity)

We could now write a FormRequest to validate this data automatically or manually validate the array using the Validator facade in the controller. To keep things simple, we'll stick to using the validator class directly:

$rules =  [
    'name' => ['required', 'string'],
    'team' => ['required', 'array'],
    'team.id' => ['required', Rule::exists('teams', 'id')],
    'team.role' => ['required', Rule::in(['engineer', 'project_manager', 'boss'])],
];

$validator = Validator::make($data, $rules);

We can now check if our data array (or "payload") is valid:

$validator->passes(); // Returns true, data is valid
$validator->messages(); // An empty message bag, great!

When passing an invalid payload array as our input, the validator will fail and provide us with a couple of validation messages:

$data = [
	'name' => 'Ruben Van Assche',
	'team' => [
		'id' => 2023,
		'role' => 'ceo'
	]
];

$validator = Validator::make($data, $rules);
$validator->passes(); // returns false!

The error bag contains the following validation messages:

[
    "team.id" => ["The selected team.id is invalid.]
    "team.role" => ["The selected team.role is invalid."]
]

Great, so up until now, everything is working as expected. We can validate both root level properties and nested properties in the team array.

Let's make things more complicated by allowing the team property to be nullable. Maybe your application allows for users without teams, so the team property can also be null.

Let's update our rules by adding the nullable rule:

[
	'name' => ['required', 'string'],
	'team' => ['nullable', 'array'],
	'team.id' => ['required', Rule::exists('teams', 'id')],
	'team.role' => ['required', Rule::in(['engineer', 'project_manager', 'boss'])],
]

When testing our two previous example payload again, everything still works as expected: the valid example payload passes and the second example contains errors so the validator fails. However, let's now test our new feature where we don't have a team at all:

[
	'name' => 'Ruben Van Assche',
]

The team property is nullable so this payload should not give us validation errors. Let's validate:

[
    "team.id" => ["The selected team.id is invalid.]
    "team.role" => ["The selected team.role is invalid."]
]

Wait what?! For some reason, the validator ran the team.id and team.role rules even though we wrote a rule that the entire team array is allowed be null, so what happened here?

The validator did its job correctly, albeit a bit bizarre. We wrote three rules for the team array. The first one tells the validator that team should be an array but can also be null. This rule succeeds with our last example payload. The other two rules require a value to be in a specific format. And these fail as the value is always required.

Laravel's validator is built in a way where each rule exists on its own. This means the two team.* rules have no context of the first rule indicating that the team array can be null. So they get executed as they would typically, resulting in two error messages.

How can we fix this?

Nullable all the way

We could rewrite our rules like this:

[
    'name' => ['required', 'string'],
    'team' => ['nullable', 'array'],
    'team.id' => ['nullable', Rule::exists('teams', 'id')],
    'team.role' => ['nullable', Rule::in(['engineer', 'project_manager', 'boss'])],
]

Now when validating the following:

[
	'name' => 'Ruben Van Assche',
]

Great! No validation messages, but what if we provide a payload like this:

[
	'name' => 'Ruben Van Assche',
	'team' => [
	    'id' => 1,
	]
]

Also no validation messages, but we expect one since a team always needs an id and role. There are better approaches to writing rules like this.

Better requiring

We could rewrite our require rules smarter by explicitly telling when a specific value is required:

[
	'name' => ['required', 'string'],
	'team' => ['nullable', 'array'],
	'team.id' => ['required_with:team', Rule::exists('teams', 'id')],
	'team.role' => ['required_with:team', Rule::in(['engineer', 'project_manager', 'boss'])],
]

When validating:

[
	'name' => 'Ruben Van Assche',
]

No messages. Up to the next one:

[
	'name' => 'Ruben Van Assche',
	'team' => [
	    'id' => 1,
	]
]

We now get the following message:

[
	"team.role" => ["The team.role field is required when team is present."]
]

Marvelous, we succeeded in writing the correct rules! But from now on, we must always use the required_with:team rule instead of the required rule and remember this for new properties to be added in the future.

We also need to be careful. If someone decides to change the name of the team property, then all our rules need to be updated. We only have two rules now, but as an application grows, these numbers tend to rise rapidly, and a mistake is quickly made.

Lastly, what if we also want to nest another array within the team array like the payload bellow, where we also want to be able to (optionally) disable or enable extra features per team:

[
    'name' => 'Ruben Van Assche',
    'team' => [
        'id' => 1,
        'role' => 'engineer',
        'features' => [
            'github' => true,
            'jira' => false,
        ]
    ]
]

And to make things more complicated, you should also be able to provide no features (in this case, defaults could be used):

[
    'name' => 'Ruben Van Assche',
    'team' => [
        'id' => 1,
        'role' => 'engineer',
    ]
]

These were the best rules I could come up with:

[
    'name' => ['required', 'string'],
    'team' => ['nullable', 'array'],
    'team.id' => ['required_with:team', Rule::exists('teams', 'id')],
    'team.role' => ['required_with:team', Rule::in(['engineer', 'project_manager', 'boss'])],
    'team.features' => ['nullable', 'required_with:team', 'array'],
    'team.features.github' => ['required_with:team.features', 'boolean'],
    'team.features.jira' => ['required_with:team.features', 'boolean'],
];

But these are not 100% watertight since the latest payload we'v constructed without the features array will result in the following validation message:

[
	"team.features" => [
	  "The team.features field is required when team is present."
	]
]

The required_with rules can fix our problem but have a few disadvantages. Is there another way?

Fixing the issue once and for all

In Flare, we don't write validation rules ourselves. One of our packages called laravel-data automatically generates them.

Let's revisit our previous example, but we now structure our data using data objects. Which has as an added benefit a stricter type structure we can use in the backend. We stop using untyped arrays with data and have objects with typed properties.

The laravel-data package can also generate TypeScript definitions based upon the types within our data objects, so our frontend code is now also typed exactly like our backend. Cool!

These are the data objects we've created based on the rules we decided on earlier:

class UserData extends Data
{
    public function __construct(
        public string $name,
        public ?TeamData $team,
    ) {
    }
}

class TeamData extends Data
{
    public function __construct(
        #[Exists('teams', 'id')]
        public int $id,
        #[In('engineer', 'project_manager', 'boss')]
        public string $role,
        public ?TeamFeaturesData $features,
    ) {
    }
}

class TeamFeaturesData extends Data
{
    public function __construct(
        public bool $github,
        public bool $jira,
    ) {
    }
}

Within our controller, we inject the User data object as follows:

class UpdateUserController
{
    public function __invoke(
        User $user,
        UserData $data,
        UpdateOrCreateUserAction $updateOrCreateUserAction
    )
    {
        $updateOrCreateUserAction($data, $user);

        flash('User updated!');

        return redirect()->back();
    }
}

As you can see, we also inject the user model we're updating and the action (a class with business logic) to perform the update within the database.

Due to the nature of laravel-data, when injected in a controller, the data object will validate the request input before creating itself. Let's take a look at the previous inputs we provided to the validator:

[
    'name' => 'Ruben Van Assche',
];

No validation messages 👍

[
    'name' => 'Ruben Van Assche',
    'team' => [
        'id' => 1,
        'role' => 'engineer',
    ],
]

Also no validation messages 🥳

[
    'name' => 'Ruben Van Assche',
    'team' => [
        'id' => 1,
        'role' => 'engineer',
        'features' => [
            'github' => true,
            'jira' => false,
        ],
    ],
]

No validation messages. We fixed our initial problem! 🎉

At this point in time, you start to wonder, is the validation working? Let's try this one out:

[
    'name' => 'Ruben Van Assche',
    'team' => [
        'id' => 1,
    ],
]

// Validation messages:

[
    "team.role" => ["The team.role field is required when team is present."]
]

It looks like everything is still validated as expected.

Internals

Let's go quickly over the internals of laravel-data. Until a few weeks ago, laravel-data worked precisely like our first approach by making all rules nullable in nested arrays. Though working, there were better solutions than this. So we completely rewrote the validation logic for laravel-data's third version.

Instead of generating rules before the payload is provided, laravel-data has a sort of JIT rule generator. It will generate rules based on the definition in the data class and the payload passed in.

For example, when we have a data object like UserData, all non-nested data property rules will be generated. The nested TeamData is nullable, so it will only start doing the same and thus generate rules for itself when we find a key team within the input payload that is not null.

If you're more interested in the internal workings of the validation rule generation, then take a look over here, where we generate validation rules for data objects just in time.

Closing

Laravel's validator is incredible, but sometimes it needs some help. Expect more articles like this on the Flare blog in the coming months and a new coat of paint for our website and application. Want a sneak peek? Mail us at support@flareapp.io.

Flare gets a lot of errors every day. To keep a good overview, we try to group them as best as possible because you don't want to end with thousands of errors in your Flare project when there's only one bug within your codebase.

For most errors, grouping is relatively easy. We look at the top frame within the call stack where the error occurred and keep track of the class, method and line where the error occurred. When Flare receives another error, we again look at this top frame and group it by these parameters.

This method works pretty well for most exceptions thrown within PHP. But there are cases where this approach isn't giving the best results.

SQL Exceptions

A Laravel application will throw a QueryException whenever something goes wrong with your database. When we would group by top frame, all database errors would be grouped into a single item. That's something we don't want since those SQL errors could be entirely different.

We could not use the top frame of the call stack. But go deeper in the stack until we find a frame within your application. But what then if an external package caused the exception? It could be that we never reach an application frame. Also, the deeper we go through the call stack, the more general the path of execution becomes. We could be grouping errors that have nothing to do with each other.

Flare uses an alternative solution. We do not group by top frame in cases of a QueryException but by exception class and message. Now, these two errors become two different items within Flare:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'titl' in 'field list' (SQL: SELECT * FROM `posts` WHERE `titl` = 'flare groupings')

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'publishe' in 'field list' (SQL: UPDATE `posts` SET `publishe` = 1 WHERE `id` = 1)

Great! But what happens to the grouping when we run this piece of code, and it fails?

public function publish(Post $post, bool $published): void
{
	$post->update(['publishe' => $published]);
}

The answer isn't that clear. It depends since a post can have multiple id's and published can be 0 or 1. The following two exceptions could be thrown when this piece of code fails:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'publishe' in 'field list' (SQL: UPDATE `posts` SET `publishe` = 0 WHERE `id` = 1)

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'publishe' in 'field list' (SQL: UPDATE `posts` SET `publishe` = 1 WHERE `id` = 1)

This will cause Flare to create two entries for errors that should be grouped since they represent the same bug. We want to avoid these scenarios at all costs!

Let's take a look at the two queries:

UPDATE `posts` SET `publishe` = 0 WHERE `id` = 1

UPDATE `posts` SET `publishe` = 1 WHERE `id` = 1

When we would use prepared statements with bindings where we sent the values separate from the query to the database server then, those queries would look like this:

UPDATE `posts` SET `publishe` = ? WHERE `id` = ?

UPDATE `posts` SET `publishe` = ? WHERE `id` = ?

Great! The two queries are identical. We can now group using the exception class: QueryException and this query with bindings. Sadly enough, often prepared statements are entirely ignored, so we cannot use them.

But what if we could strip all values from these queries to look like queries with bindings?

Parsing queries

SQL queries are complicated, we've seen some simple queries earlier, but they can become quite complex. That's why we'll use a SQL parser to parse the queries into tokens so we know what should be replaced and whatnot.

The first thing we need to do is parse the query:

(new PHPSQLParser($query))->parsed;

Such a parsed query looks like this:

array:3 [
  "UPDATE" => array:1 [
    0 => array:10 [
      "expr_type" => "table"
      "table" => "`posts`"
      "no_quotes" => array:2 [
        "delim" => false
        "parts" => array:1 [
          0 => "posts"
        ]
      ]
      "alias" => false
      "hints" => false
      "join_type" => "JOIN"
      "ref_type" => false
      "ref_clause" => false
      "base_expr" => "`posts`"
      "sub_tree" => false
    ]
  ]
  "SET" => array:1 [
    0 => array:3 [
      "expr_type" => "expression"
      "base_expr" => "`publishe` = 0"
      "sub_tree" => array:3 [
        0 => array:4 [
          "expr_type" => "colref"
          "base_expr" => "`publishe`"
          "no_quotes" => array:2 [
            "delim" => false
            "parts" => array:1 [
              0 => "publishe"
            ]
          ]
          "sub_tree" => false
        ]
        1 => array:3 [
          "expr_type" => "operator"
          "base_expr" => "="
          "sub_tree" => false
        ]
        2 => array:3 [
          "expr_type" => "const"
          "base_expr" => "0"
          "sub_tree" => false
        ]
      ]
    ]
  ]
  "WHERE" => array:3 [
    0 => array:4 [
      "expr_type" => "colref"
      "base_expr" => "`id`"
      "no_quotes" => array:2 [
        "delim" => false
        "parts" => array:1 [
          0 => "id"
        ]
      ]
      "sub_tree" => false
    ]
    1 => array:3 [
      "expr_type" => "operator"
      "base_expr" => "="
      "sub_tree" => false
    ]
    2 => array:3 [
      "expr_type" => "const"
      "base_expr" => "1"
      "sub_tree" => false
    ]
  ]
]

We only want to replace the constants within the query, so we update each token where the expr_type is const to not use a value like 1 or 0 but to use a ? just like with the bindings in prepared statements:

array:3 [
  "expr_type" => "const"
  "base_expr" => "1"
  "sub_tree" => false
]

Becomes:

array:3 [
  "expr_type" => "const"
  "base_expr" => "?"
  "sub_tree" => false
]

The cool thing with this parser package is that we can convert these tokens back to a SQL query:

(new PHPSQLCreator())->create($parsed);

And we're done! Our query now looks like this:

UPDATE `posts` SET `publishe` = ? WHERE `id` = ?

Within Flare, the algorithm of stripping the token tree is a bit more complex than we show here, but the gist is the same. Take some raw values and replace them with question marks.

More exotic queries

But this is not where we end, although the parser is quite good at parsing the wildest queries. Sometimes it just cannot understand what's happening. For example:

UPDATE users SET uuid = 89637150-f807-11ea-aee4-51fe58f2b2b3

In some database configurations, this query is valid. And our parser will only strip the first part of the UUID because it thinks the - signs are operators, and the other parts of the UUID are references to columns within your table:

UPDATE users SET uuid = ? - f807 - 11ea - aee4 - 51fe58f2b2b3

The same thing happens with dates:

UPDATE users SET date = ? - ? - 2020T17:24:34.012345Z

That's why, before we parse the query, we preprocess it by stripping out some well-known types of strings like dates, emails, UUIDs:

preg_replace(
	[
		'/\S+@\S+\.\S+/', // emails
		'/\d{2,4}-\d{2}-\d{2,4}[ T]?(\d{2}:\d{2}:\d{2}([+-]\d{2}:\d{2})?(\.\d{6}Z)?)?/', // dates
		'/[0-9a-f]{8}\b-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-\b[0-9a-f]{12}/', // uuids
	],
	['?', '?', '?'],
	$query
);

Preprocessing makes the queries a bit more readable for our parser, which improves our final result drastically!

Connection issues

A QueryException always looks like this:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'publishe' in 'field list' (SQL: UPDATE `posts` SET `publishe` = 0 WHERE `id` = 1)

Did you notice the SQLSTATE[42S22] part? It is a code telling you what went wrong with your database. For example, a connection issue in a MySql database has the following code: [HY000][2002] looks familiar?

Within Flare, we'll also take a look at these codes. When we notice the error is caused by connection issues and not a bug within your code, then we group all these errors since your database server was probably just a few moments down.

Impossible queries

Let's take a look at a final example:

SELECT * FROM invoices WHERE number = 123ABC

Such a query is a difficult one. The parser doesn't know if 123ABC is a constant or a reference to a column within your database table.

In the latter case, we don't want to replace the column name with a ? since these are not values that change between exceptions.

In such cases, we do not strip the value and keep that part of the query.

class UserResource extends JsonResource
{
    use HasLinks, HasMeta;
    
    public function toArray($request): array
    {
        return [
            'id' => $this->id,
            'name' => $this->name,
            'email' => $this->email,
            'links' => $this->links(UsersController::class),
        ];
    }

    public static function meta()
    {
        return [
            'links' => self::collectionLinks(UsersController::class),
        ];
    }
}

In a response this would become the following Json:

{
   "data":[
      {
         "id":1,
         "name": "Ruben Van Assche",
         "email": "ruben@spatie.be"
         "links": {
            "show": "https://laravel.app/users/1",
            "edit": "https://laravel.app/users/1/edit",
            "update": "https://laravel.app/users/1",
            "delete": "https://laravel.app/users/1"
         }
      }
   ],
   "meta": {
      "links": {
         "index": "https://laravel.app/users",
         "create": "https://laravel.app/users/create",
         "store":  "https://laravel.app/users"
      }
   }
}

It was actually a url generator for lazy developers.

Why abandoning the package?

Allthoug the package still works, and I'll guess as long as Laravel doesn't change too much about the routing systsem it will still work for quite a while we decided to stop the development.

Originally this package was built for one project, and it is still being used there. But since then we've created many (much bigger) projects then that one project and tought we could use this package. But we didn't.

The idea initially sounded great, never write links in resources again. But it is actually not that much work to write links, if we rewrite the above resource for example:

class UserResource extends JsonResource
{
    use HasLinks, HasMeta;

    public function toArray($request): array
    {
        return [
            'id' => $this->id,
            'name' => $this->name,
            'email' => $this->email,
            'links' => [
                "show" => action([UsersController::class, 'show'], $this),
                "edit" => action([UsersController::class, 'edit'], $this),
                "update" => action([UsersController::class, 'update'], $this),
                "delete" => action([UsersController::class, 'delete'], $this),
            ],
        ];
    }

    public static function meta()
    {
        return [
            'links' => [
                "index" => action([UsersController::class, 'index']),
                "create" => action([UsersController::class, 'create']),
                "store" => action([UsersController::class, 'store']),
            ]
        ];
    }
}

That's not so bad, it's a bit more code but a lot more readable. Adding or removing certain links from a resource is now super easy. Do you want to add an array with some extra endpoints within the links property? Not a problem! Do you want use other names for the link properties? Change them! Do you want to add an action to another controller? Do it!

Although the package supported all these features, they weren't always that easy to use. Almost always when adding links you had to check the docs if you wanted to do something little more complicated than just adding a link to a controller.

That brings us to the next reason for abandonment, magic! Automatically generated links sounded great in the beginning but they are a big black magic box you don't want to open. Most of the time the package worked as expected. But there were cases where it didn't and debugging those cases was hard. Especially those cases where you had resources within resources, the horrors!

The package replaced some parts of the Laravel routing system so it could plug into it and also did some crazy stuff with the Laravel Resources, they actually weren't built for this kind of stuff.

I haven't touched the code for two years and I actually have no idea what it does and how it works. There is a class ParameterResolver that's a 130 lines, it tries to find parameters for constructing the links and I'm really scared of it. That's not a good sign.

My first though was to write a second version, a more light version of the package. But that would still require a lot of crazy code to interface with Laravel. Why would we?

Conclusion

That's it, farewell laravel-resource-links, you were the first package I've ever made, since then I've learned so much about package development, but your time has come.

If you don't want to manually change all your links within your application take a look at Ziggy it works a bit different then laravel-resource-links but also has the ability to automatically generate links.

spatie/one-package-to-rule-them-all is probably the biggest package we've ever built at Spatie. That's because it includes every package built at Spatie to date.

The coffee breaks required when composer was installing new dependencies are gone

Over are the days, you had to add a Spatie package to composer manually. Now, you include spatie/one-package-to-rule-them-all in your project, and you have all the Spatie packages available to you. It's not possible anymore to make a mistake between `installing` and `requiring` a composer package. The coffee breaks required when composer was installing new dependencies are gone, productivity will rise again.

How does it work?

The most crucial file within the package is the composer.json. I think we really did something groundbreaking here. Usually, we're thinking hours about the required dependencies a package needs to find the most minimal set of dependencies. This time we decided to take a different approach. Why add only a few dependencies if we could add every Spatie package available?

Though the idea is simple, the execution was absolute hell. Some of our older packages require PHP 7 or even PHP 5.6 ewww. We wanted to make this package a first-class PHP 8 citizen, but that would mean we couldn't include the older packages with unsupported PHP versions.

Luckily after a few long coffee breaks with the whole team, we've found the solution. Instead of requiring these older packages, we can simply suggest them to you if you want to install them.

I think this package is going to change the Laravel landscape forever. Try it out by once and for all installing the package:

composer require spatie/one-package-to-rule-them-all

And let us know what you think!

PS: do not forget to run the php artisan package:inspire command

In the long run, id's have only one practical use: identifying things, and with things, I mean everything. An id can be used to represent a user in your database. A drive on your computer. Or a transaction from your bank. An id can represent god knows what resource.

Id's can identify everything around us, which's its strongest point and its greatest downfall. Do you know what id = 5 or UUID = 00145851-4612-4513-82ca-68b77df0a0b0 represents? Though the id can point to a specific resource, it does not tell you what kind of resource it is.

Stripe solves this by prepending the id's with a type it is representing. A customer id will always start with cus_, and a charge always has ch_ prefixed. The id does not only identify a specific resource. It also tells you what kind of resource it is.

I'm in love with this concept. Every time you're working with id's within your application. You know the kind of resources you're working with. But wouldn't it be cool if your programming language and IDE also knows which resources these id's represent?

Let's take a look at a simple blogging application where a user can comment on a post. I'm using Laravel in this example, but you can use this method in every framework or programming language.

At some point you want to write a piece of code to add a new comment to a post, the signature of such piece of code may look like this:

public function addComment(
	int $userId,
	int $postId,
	string $content
): Comment;

Now comes the tricky part, what if within a few months we decide to switch the order of the id's because it feels more natural:

public function addComment(
	int $postId,
	int $userId,
	string $content
): Comment;

We'll need to update all these method calls throughout our codebase, so the code keeps working. What if we forget one? At some point, the id of a post will be given as user id and vice versa. These kind of errors are difficult to spot and take a lot of time debugging.

There are solutions to this problem, we could for example always use models as a parameter to the function:

public function addComment(
	Post $post,
	User $user,
	string $content
): Comment;

When we provide a User model as the first parameter, PHP will complain because the types are not the same. We get an exception and don't need to debug for hours to find the problem, neat!

But, we always need to have the Post and User models loaded when we want to call this method. Hydrating models can be quite resourceful, especially when you've got a lot of models. What if we want to represent id's not tied to models, such as the Stripe resources we saw earlier? We don't have models for each Stripe charge or customer, so we cannot use them as a type for our method.

Let's go back to where we started. The only thing we want is typing our id's so we can pass them to functions correctly. What if we put these id's, the strings and integers in their own object? It's a little bit of both worlds, we don't need models and can represent non-models, but we keep the strong typing:

class PostId implements Stringable{		
	public function __construct(
		private int $id
	){}
	
	public function id(): int
    {
		return $this->id;
	}
}

When we also create such id for the User model we can now refactor out initial piece of code to this:

public function addComment(
	PostId $postId,
	UserId $userId,
	string $content
): Comment;

Great, passing a UserId as a PostId will cause an exception! Since we'll have a lot of these id classes, let's create a base class for them:

abstract class Id implements Stringable{
	public static function create(int $id): static
    {
		return new static($id);
	}

	public function __construct(
		private int $id
	){}
	
	public function id(): int
    {
		return $this->id;
	}
	
	public function equals(int|Id $other): bool
    {
		return (string)$this === (string)$other;
	}
	
    public function __toString(): string
    {
        return $this->id;
    }
}

Now we let PostId and UserId extend from this Id class, which gives us the ability to create a new id and easily compare them.

Since these id's represent models, we can take this even a bit further in a Laravel application. Let's create yet another abstract class: ModelClass:

abstract class ModelId extends Id{
    abstract protected function getModelClass(): string;
    
	public function __construct(
		private int $id
	){}

    public function model(): Model
    {
        return $this->modelClass::find($this->id);
    }
    
    public function exists(): bool
    {
    	return $this->model() !== null;
    }
}

We let PostId and UserId extend from ModelId and need to implement the abstract function getModelClass:

class PostId implements Stringable{		
	public function getModelClass(): string
    {
		return Post::class;
	}
}

Now we can do the following:

$postId = PostId::create(10);

When we can easily check if the model exists:

$postId->exists(); // true

Or get the model itself:

$postId->model(); // Post object

This is only useful for id's representing models. The Stripe id's we talked about in the beginning would only extend from the Id class since they're not models.

We don't have to stop here! In our project, we've added some extra functionality to these abstract classes like:

• creating multiple id's in one go
• the ability to pass null to the create function, which will return null instead of an Id object
• a simple helper to get the morph class from a ModelId

Let's go full circle. We can easily make a new Id with the create method, but wouldn't it be cool if we could get the Id from a model like this:

$post->getId();

We create a new interface which the Post and User models will implement:

interface WithId{
	public function getId(): Id
}

Now we add this interafce to our models as such:

class Post extends Model implements WithId{
	public function getId(): Id
    {
		return new PostId($this->id);
	}
}

And we're done! As a bonus, since our Id class is Stringable, we can still use the Eloquent functions that Laravel provides, for example, this will work:

$postId = PostId::create(20);

$post = Post::findOrFail($postId); //Post object

When you're writing big applications, encapsulating id's can make your code a lot more readable for everyone working on the project. Give it a try in your next project and let me know what you think about it!

But sometimes you've got to work on an older project. When you want to switch between PHP versions in your terminal and Laravel Valet it is only one terminal command away:

valet use php@7.4 --force

Want to switch back to PHP 8 then use php@8.0. Or just php since it's the most recent released PHP version at this moment. You can switch to any PHP version installed with Homebrew on your system:

valet use php@8.0 --force
valet use php@7.4 --force
valet use php@7.3 --force
valet use php@7.2 --force
valet use php@7.1 --force
valet use php@7.0 --force
valet use php@5.6 --force

You'll probably need the shivammathur/homebrew-php tap to install some of these older versions with Homebrew.

Fixing a stuck Valet PHP version

Sometimes, the valet use command only switches the PHP's terminal version, but Valet's PHP version keeps stuck. It can be that the valet socket is not configured properly, removing it normally solves the problem:

rm ~/.config/valet/valet.sock
valet restart

When we want to link a comment to a post, we have to create a new Comment model with commentable_id equal to the id of the post and commentable_type equal to App\Post, the class of the post.

But what if we change the location of our Post model from App\Post to App\Models\Post? In our database, the commentable_type fields won't be updated, and thus Laravel cannot find the comments associated with our post since the App\Post model doesn't exist anymore.

Yes, we could add a migration to update the values of commentable_type with the correct post model, but that's quite a bit of work that can be quickly forgotten.

Laravel has an excellent solution to this: morph maps! Within your application, you register a mapping between the class of your model and a simple identifier. In our scenario, the morph map would look like this:

Relation::morphMap([
    'post' => 'App\Post',
]);

In the database, we will now use post in the commentable_type row instead of App\Post. Whenever we want to move our Post model, the only thing we have to do is updating the morph map, neat!

Maintaining morph maps requires discipline

I cannot count the number of times I forgot to add an entry to a morph map, and the full class name of a model would be stored in the database.

We're working on a massive project these days with lots of models. And I know I'm going to forget to add some models to the morph map. My colleagues will forget to add some of these models. I think everyone sometimes will forget to add a model to the map.

Can't we do better? Is it possible to automatically generate a morph map, so we don't forget to add these models?

Dynamically generating morph maps

Okay, all we have to do is add a method to a model, so we know its morph identifier. Luckily, each Eloquent model already has a method getMorphClass, which Laravel uses to save the model's morph identifier.

getMorphClass will return the model's full class name when the model is not present in the morph map. When the model exists in the morph map, its identifier from the map will be returned. We overwrite this function in each model like this:

class Post extends Model
{
    public function getMorphClass()
    {
        return 'post';
    }
}

Cool! We're halfway done. When we save a comment linked to a post, it will use post as commentable_type and not the full class name.

But a morph map is still required when we want to load the post associated with the comment since Laravel doesn't know that post is an App\Models\Post model.

Creating such a map is now actually really simple. We can search for all the models in our application, run the getMorphClass method on them. And we're done. We've got ourselves a dynamically generated morph map!

Making sure every model has a morph identifier

We can even take it a step further. It is still possible that someone forgets to add an implementation of getMorphClass to a model. Let's fix that!

We'll make an abstract base model from which all our models will extend:

abstract class Model extends BaseModel
{
    public function getMorphClass()
    {
        $class = get_class($this);

        throw new Exception("Model `{$class}` hasn't implemented `getMorphClass` yet");
    }
}

Now when a model doesn't implement getMorphClass, an exception is thrown. And we know for sure each model has a morph identifier.

This won't create a lot of exceptions in production. Since at the boot process of our application, getMorphClass will be called for each model. In our local setup, we'll know which models are missing a getMorphClass implementation immediately.

Cool, we're done now we've got ourselves a dynamically generated morph map!

Package it up!

I've added this functionality into a new package called: laravel-morph-map-generator, which adds some extra functionality like caching dynamically generated morph maps on production, ignoring models, and more!

Let me know what you think about it.

At Spatie, we fell in love with Inertia. The framework makes it so easy to create a backend Laravel application with a React frontend. Passing data between these sides is so straightforward. You can give them to a React component just like you would give them to a regular Blade view.

What made us fall in love with this setup is that we're now able to use TypeScript, which made our Javascript code type-checked and our frontend code less error-prone.

But with this type-checking, a new problem popped up. What about structures that would have to be used on both sides of the application? Take, for example, an enum like this one:

/**
 * @method static self guitar()
 * @method static self piano()
 * @method static self drums()
 */
class Instrument extends Enum
{
}

This enum lives on the backend because we're going to need it to save a model, validate a request, or make some decisions further in the process. But what happens if we would need this enum on the frontend?

This enum's values can be communicated quite easily, get all the options(guitar, piano, and drums) and put them in an array with a corresponding label. Then we provide this array to the react component via Inertia at the end of our controller:

class MusiciansController
{
    public function show(Musician $musician)
    {
        return Inertia::render('Musician/Show', [
            'musician' => $musician,
            // the array we created
            'instrumentOptions' => Instrument::options(), 
        ]);
    }
}

On the frontend in our React component, we now have the following array with instruments:

[
	{'label': 'guitar', 'value': 'guitar'},
    {'label': 'piano', 'value': 'piano'},
    {'label': 'drums', 'value': 'drums'}
]

We want to show a select component where the musician can select its favorite instrument. We can pass this array to such a component, and we're done!

That's true for simple forms, but what about a situation where you want to show a checkbox to mark you are playing bass guitar when choosing guitar as an option?

That would require a condition like this:

if(instrument === 'guitar'){
	// Show checkbox
}

But actually, what's the type of this instrument variable? It's a string coming from the backend, and our frontend code doesn't know much more about it. We know this is no ordinary string. It's an enum of type Instrument, which means the string can only have three values: piano, guitar, and drums.

Luckily TypeScript can help us here. We can create an Instrument type:

export type Instrument = 'piano' | 'guitar' | 'drums';

When we now assign the instrument variable to value violin. The TypeScript type checker would then complain since instrument can only be set to piano, guitar, or drums.

This kind of type checking is cool! But there's a big catch. We're writing out types two times: one time in PHP and one time in TypeScript.

What will happen if we remove the guitar entry from the enum in PHP? It would still be in the TypeScript definition. Wich can break our code or give us incorrect type errors. Or what if we add a violin to our PHP Enum? This can go bad very quickly.

In small projects where one or two people are working on, this isn't such a big problem. You can communicate with your colleagues about these changes if the amount of types is small.

In this project, we worked with seven people, and these seven people would sometimes not work on the project for weeks. There we're backend developers who wouldn't write TypeScript and frontend developers who wouldn't write PHP. Even worse, we knew the number of types grow dramatically in the timespan of the project.

There were three options now:

1. Type nothing; this is the worst option and was a no go for us.
2. Try to keep these definitions in sync, but since this project was so big and we were working on it with many people, this was going to get out of hand.
3. Try to keep these definitions in sync automatically.

Introducing Typescript Transformer

We've made a package for that third option, it's called laravel-typescript-transformer, and it will convert all the types from the backend that are needed in the frontend. The package will try to transform these types to TypeScript automatically. Let's take a look at it!

All we have to do is adding an @typescript annotation to our Instrument enum:

/**
 * @method static self guitar()
 * @method static self piano()
 * @method static self drums()
 * 
 * @typescript
 */
class Instrument extends Enum
{
}

Now when running:

php artisan typescript-transform

The package will create a generated.d.ts file in your Laravel resources directory containing:

export type Instrument = 'piano' | 'guitar' | 'drums';

Cool! When we add or remove or add an instrument from the enum, and this type of instrument is being used in our code, then the TypeScript checker will notice this and complain. This checker is running almost always, whenever someone is working on the frontend code. Or when we commit code, then a GitHub Action will also check the code, neat!

Our initial case was to transform enums and keep them in sync. This was so powerful we didn't stop there!

Let's say you have a data transfer object(DTO), an object with some public properties you can pass around in your application. In the past, we had to keep a type definition for this DTO in PHP and TypeScript. Wouldn't it be cool if we could have an automatically generated TypeScript definition for that object?

All you have to do is add the annotation:

/** @typescript */
class MusicianData extends DataTransferObject
{
    public string $name;
    
    public int $age;

    public Instrument $instrument;

    public static function create(array $data): self
    {
        return new self([
            'name' => $data['name'],
            'age' => $data['age'],
            'instrument' => Instrument::make($data['age'])
        ]);
    }
}

Now when running the Typescript Transformer command, our generated TypeScript file looks like this:

export type Instrument = 'piano' | 'guitar' | 'drums';

export type MusicianData = {
    name : string;
    age: number;
    instrument: Instrument;
}

We use this DTO for two purposes:

• When we would give the Musician model to a form in the frontend, so we can build a form for creating/editing the model
• When that form is filled in, and we receive the data from the frontend, which we will use in the backend

In our project, we stopped using the default Laravel resources in favor of DTO resources that also could be typed on the frontend. In the package documentation, we've added a whole section on how you could accomplish this.

We even use these resources to log activity on models, which is very powerful but something for another blog post.

How does it work?

In essence, the process of converting PHP classes to Typescript is relatively easy. Let's take a look at the steps:

Searching for PHP classes that need to be converted
The package will look in a directory you defined in your config for classes with a @typescript annotation and make a list. It is possible to add classes to that list without @typescript annotation automatically, you can read more about it here.

Finding a transformer for the class
Transformers are the heart of the package. They will take a PHP class and output a TypeScript representation. We've included some essential transformers in the package, but you probably also want to write your own transformers.

Replacing missing symbols
Each transformer will output a TypeScript representation, but what about types that link to other types? For example, in our MusicianData, we link to the Instrument enum. At the time of transforming, we do not know what the TypeScript type of Instrument will be, so we cannot add the type to the MusicianData type (yet).

That's why each transformer will keep track of a list of links to other types it does not know yet. In this step, we replace these missing links with the correct types.

Write out the generated TypeScript
In this last step, we'll write down the transformed types into one TypeScript file that you defined in your config.

That's it! Although this transformation looks relatively straightforward, there's a lot more to look at. For example, you can create inline types, or use another name for the TypeScript type or completely modify the types of the properties of DTO's as you wish.

Be sure to take a look at the laravel-typescript-transformer package. We've also made a typescript-transformer package that's not Laravel specific, which is the basis for the Laravel package and can be used in any PHP project.

The future

Although the initial version is just released, I've already some things on my list for the second version of the package:

• A watcher which automatically transforms types as they are changed
• Support for nullable TypeScript types
• Support for omitting namespaces in TypeScript types

Disclaimer: If you haven't read my previous post, please, read it first! It explains the basic principles of Actions which we continue to use in this post.

Welcome to the matrix 🔴-🔵

Let's get started and continue where we left the previous time: creating a testing workflow. First, we create a new workflow file in our repository:  .github/workflows/testing.yml and start with a basic template:

name: Test

on: [push]

jobs:
    test:
        runs-on: ubuntu-latest

Time for the first difference between application and package testing, let's add a strategy key to the test section:

test:
    runs-on: ubuntu-latest
    strategy:
        fail-fast: true
        matrix:
            php: [7.2, 7.3, 7.4]
            laravel: [5.8.*, 6.*]
            dependency-version: [prefer-lowest, prefer-stable]

Wow! There's quite a lot happening here. Let's go through it step-by-step. What we're trying to achieve is matrix testing, simply said: we try to run our tests on a lot of different configurations so we can be sure that our package is working in different environments.

Each configuration will spawn a GitHub Action job with specific configuration options. GitHub Actions will inject the configuration options for each configuration as variables so we can use them through the workflow. In this case, the configuration options are: the PHP version, the Laravel version, and if we desire our composer dependencies to be stable or the lowest possible.

An example configuration could look like this:

• PHP 7.4
• Laravel 5.8.*
• prefer-stable dependencies

In our workflow under the strategy section we create a matrix section, here we can specify the configuration options by giving them a key and an array of options from which GitHub Actions will create configurations.

When this workflow runs, it will spawn a total of 12 jobs! You can calculate this as such:

3 PHP versions (7.2, 7.3, 7.4) * 2 Laravel versions (5.8.*, 6.*) and 2 composer flags (prefer-lowest, prefer-stable) = 12.

Next to the matrix key in the strategy section we've added that fail-fast will be true. This setting is not required but becomes quite handy when you're spawning a lot of jobs. If one of your configurations fails, then all the other configurations of that workflow will immediately fail. This will reduce the number of jobs running without purpose because they will also probably fail.

When testing a Laravel package, we also going to need Testbench. The problem is Laravel 5.8.* requires Testbench 3.8.* and Laravel 6.* requires Testbench 4.*. We can add an include section to our matrix in which we simply say: "if this variable is this value, then add another variable with this value." In our workflow, it looks like this:

matrix:
    php: [7.2, 7.3, 7.4]
    laravel: [5.8.*, 6.*]
    dependency-version: [prefer-lowest, prefer-stable]
    include:
        -   laravel: 6.*
            testbench: 4.*
        -   laravel: 5.8.*
            testbench: 3.8.*

Using matrix variables

So we defined our configurations, now in every job, we will have access to the following variables: php, laravel, dependency-version, and testbench. Let's try this out! We're going to give our job a name with the variables of our configuration. This can be done by adding a new key name in the test section:

test:
    runs-on: ubuntu-latest
    name: PHP ${{ matrix.php }} - Laravel ${{ matrix.laravel }}
    strategy:
        ...

As you can see the configuration variables are scoped by the matrix keyword, with our example configuration from above the name of the job will look like: PHP 7.4 - Laravel 5.8.*

Stepping through the matrix

Everything is set up let's start testing! First we will be defining the steps that will run in each job. We start with checking out the code from GitHub:

-   name: Checkout code
    uses: actions/checkout@v1

Then we'll install the PHP version of our configuration:

-   name: Setup PHP
    uses: shivammathur/setup-php@v1
    with:
        php-version: ${{ matrix.php }}

It is time for the next difference between application and package testing with Actions! In the application workflow, we used the build-in PHP version of Actions, which at the time of writing is 7.3. If we want to matrix test our PHP version, then we have to set it up. This can be done by using the setup-php action. It takes the php-version as an option which we fill in with our configuration variable ${{ matrix.php }}.

Want to add some extra extensions to PHP or change some ini values? Then check the GitHub page of the setup-php action for more information.

Next, we install the composer dependencies:

-   name: Install dependencies
    run: |
        composer require "laravel/framework:${{ matrix.laravel }}" "orchestra/testbench:${{ matrix.testbench }}" --no-interaction --no-update
        composer update --${{ matrix.dependency-version }} --prefer-dist --no-interaction --no-suggest

We first require a specific Laravel and Testbench version before installing our other dependencies with the prefer-stable or prefer-lowest composer flag.

Of course, we can also cache our dependencies, so the next time the Action runs it will even run faster. In the application workflow, we had access to a composer.lock file that we used as a key to identify if the cache entry existed. But we don't commit a composer.lock file to the repositories of packages.

The solution is to use the hash of the composer.json file and the configuration variables. The step will in the end look like this:

-   name: Cache dependencies
    uses: actions/cache@v1
    with:
        path: ~/.composer/cache/files
        key: dependencies-${{ matrix.dependency-version }}-laravel-${{ matrix.laravel }}-php-${{ matrix.php }}-composer-${{ hashFiles('composer.json') }}

Do not forget to put this step before the Install dependencies step. Otherwise your cache would not be used.

What if one of the packages in composer.json get's bumped with a new version? We would expect that our cache will be invalidated, but that's not going to happen 😞. The only times our cache will be invalidated will be when our composer.json file changes. Now, this is not that big of a problem. Every week GitHub purges the caches and Composer will always use the latest dependencies possible even if they are not cached.

Now for the last step of the workflow, we run our tests:

-   name: Execute tests
    run: vendor/bin/phpunit

If everything is going well, your workflow should look like this:

Badges

Who doesn't like badges? You can create them for your GitHub Actions. Head over to shields.io and search for: GitHub Workflow Status.

Conclusion

This powerful yet straightforward workflow works for 90% of our packages. Matrix testing provides us the trust that our packages run on the environments we defined. You can find the workflow we've build here.

If you want to read more about testing packages, then head over to Freek's blog, where he adds Github Actions to Laravel Ignition. In the next part, we will take a look at creating our own action for the checking and fixing of our PHP styling. See you then!

With GitHub Actions, you can spin up a container in no time, use actions from other authors to run in your container, commit to your repositories, and that all without leaving GitHub. And even better, the free tier GitHub provides you is very generous. With a free account, you can spin up to 20 containers concurrently and run 2000 minutes of containers for free.

Using GitHub actions starts with workflows you define in your repository, which consist of jobs that will be run concurrently, and in these jobs, you can define a sequence of steps that should be executed. A workflow will be triggered by an event like a push, pull request, or even a cron you can define.

You can write workflows in YAML, which makes them easy to write and read. In the beta version of GitHub Actions you had to use Ocaml, which was quite hard to comprehend, and there was almost no documentation. If you were a bit frightened by the beta version, like me, then rest assured: the YAML version is easier to use, and the documentation is well written.

Creating a test workflow

Let's say we want to run our test suite each time someone commits to our repository. First, we create a tests.yml workflow file in .github/workflows:

name: Tests (PHP)

on: [push]

jobs:
    tests:
        name: Run tests
        runs-on: ubuntu-latest
        steps:
            -   uses: actions/checkout@v1

This is the most basic workflow we can have. First, we define by the on key that this workflow will run when a commit was pushed. Then a job named Run tests is created which will run on the latest version of Ubuntu, it is possible to use other machines like Windows or macOS.

The last thing we do is defining the steps this job consists of. In this example, this is just one step: we checkout the code of the commit that was pushed. A step can be an action that was defined by you or someone else, and that will run some code. You can also immediately run code in the shell of the container.

In this case, the step is an action created by GitHub itself, we import it with the uses keyword. Actions can be imported from folders within your project or from GitHub, just like the checkout action.

Let's add some steps and go through them step-by-step! A small note, within the examples below I've changed the indentation a bit for readability, don't forget to indent it back correctly when creating your workflow!

Running composer‌

-   name: Run composer install
    run: composer install -n --prefer-dist
    env:
        APP_ENV: testing

Downloading and installing dependencies via composer will be the first thing we do. The uses keyword is now changed to run so we can execute commands directly in the container. There's also an env key, here you can pass in variables like in your Laravel .env file, neat!

Preparing Laravel‌

-   name: Prepare Laravel Application
    run: |
        cp .env.example .env
        php artisan key:generate

Next we create our Laravel application by making an .env file from the .env.example and by setting an application key. The pipe on the first line of the run keyword makes it possible to execute multiple lines of commands.

Running Yarn‌

-   name: Cache yarn dependencies
    uses: actions/cache@v1
    with:
        path: node_modules
        key: yarn-${{ hashFiles('yarn.lock') }}

-   name: Run yarn
    run: yarn && yarn dev

Two steps? We are running yarn in the second step, a simple action by now that will create a node_modules directory with the dependencies of the project. The first step will cache the node_modules directory. This is specified by path within the with key of our step. We will cache the node_modules directory as long as the hash of the yarn.lock file stays the same, we can specify this with key.

The with key of an action allows you to configure how actions run, you can read about these parameters in the readme's of actions. The ${{ }} syntax allows you to run code within your YAML file. You can read more about it here.

We can also cache the composer dependencies installing, instead of caching the node_modules directory we cache the vendor directory. We can write this action as such:

-   name: Cache composer dependencies
    uses: actions/cache@v1
    with:
        path: vendor
        key: composer-${{ hashFiles('composer.lock') }}

-   name: Run composer install
    run: composer install -n --prefer-dist
    env:
        APP_ENV: testing

Testing our application

We've prepared our container for running tests, and now the time has come to do that, let's have a look at the step:

-   name: Run tests
    run: ./vendor/bin/phpunit
    env:
        APP_ENV: testing

Nothing fancy here: we run PHPUnit from the vendor dir and set the environment to testing. Our test suite will use the env variables set by the phpunit.xml file:

        <env name="APP_ENV" value="testing"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="SESSION_DRIVER" value="file"/>
        <env name="QUEUE_CONNECTION" value="sync"/>
        <env name="DB_CONNECTION" value="sqlite"/>
        <env name="DB_NAME" value=":memory:"/>
        <env name="DEBUGBAR_ENABLED" value="false"/>

But you're free to change these variables in your action, for example, you could change the type of database connection to MySQL as such:‌

-   name: Run tests
    run: ./vendor/bin/phpunit
    env:
        APP_ENV: testing
        DB_CONNECTION: mysql
        DB_NAME: our-awesome-tested-app
        DB_PASSWORD: 
        DB_USER: root

The last thing we do is storing any logs created by our Laravel application when the tests crash, this can be achieved by using the upload artifacts action:‌

-   name: Upload artifacts
    uses: actions/upload-artifact@master
    if: failure()
    with:
        name: Logs
        path: ./storage/logs

As you can see, we've added a new keyword to our step: if. This will make this action only run if a certain condition is true. In this case, this action will only run if the previous action failed due to the failure() expression, there are a lot of other expressions that you can use, for example, the always() expression will always run your step even if the previous steps failed.

Running the workflow

Now it's time for the big moment! Commit and push this workflow and head over to your repository in GitHub, in the `Actions` tab,  you will see a container getting started, and it will go through all our steps!

Creating a mysql database in the job

In the example above, we used an SQLite database for testing, this is extremely fast but has some disadvantages like no support for JSON columns. You can also use a MySQL database, but it requires a bit more setup.

We're going to pull in a MySQL docker image, this will start a MySQL server in our container. We can do this by adding a services key to the tests entry like so:

// ...

jobs:
    tests:
        // ...

        services:
            mysql:
                image: mysql:5.7
                env:
                    MYSQL_ALLOW_EMPTY_PASSWORD: yes
                    MYSQL_DATABASE: our-awesome-tested-app
                ports:
                    - 3306
                options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
                
        // ...

Within the services section of the workflow, we can define services that will run in the background during the execution of the workflow. You could, for example, also add a Redis server here.

We add a new MySQL service to the services section. For this service we will be using the mysql:5.7 image from docker. This is an official MySQL docker image that can be configured. We configure the creation of a new database called our-awesome-tested-app and allow the root password to be empty in the env section, the port for the MySQL database will be 3306 by setting it in the ports key.

The last section in our MySQL service is the options key. Here we can define some specific options to the docker service. In this case, we will wait until the MySQL server responds so we know for sure it is running.

Conclusion

GitHub Actions are a welcome addition to every developer's toolbelt, I will be using it for all of my next projects. Within the next few weeks I will try to publish some more posts about creating your own actions, using a matrix to test multiple PHP versions and a workflow to lint and fix your PHP/Js code. See you then!

Want the whole workflow file in one piece? You can find it here.